Member since
11-29-2022
2
Posts
0
Kudos Received
0
Solutions
01-19-2023
07:21 AM
hello. I'm having a problem with Nifi, so I'm posting a question. In my Nifi, an error occurs in one node, and there is a problem that the node is disconnected in the cluster A total of 6 nodes were configured as a cluster with 3 nodes in different datacenters. The current configuration is collecting network logs through ListenSyslog, ParseCEF, and then sending them to graylog. It receives an average of 721MB per 5 minutes, but there is a problem that the cluster becomes Disconnected with a Queue Full error occurring on some nodes. The error message is below. Communication Failed with Remote Address[null]: java.lang.illegalStateException: Queue full It says Queue full, but I don't know what kind of queue the error message points to and how to solve it. My nifi.properties configuration are below. nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=800000
nifi.queue.backpressure.size=10 GB
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.queue.swap.threshold=100000
nifi.provenance.repository.index.shard.size=8 GB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min The servers have specifications of 48 cores (HyperThread enable) and 256GB Memory. please help me thank you.😭
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
11-29-2022
06:12 AM
Hi, I want ask a shared variable between non-associated Processors. I have 2 separate Process Groups. One group monitors the status of the API called log manager, and changes the variable 'Circuit_Break' to 'enable' through UpdateAttribute when the API status is bad. Another group receives syslog as udp and put it in an API called log manager. At this time, if the variable called 'Circuit_Break' is 'enable' through RouteOnAttribute in the middle, alert with slack and try to execute ControlRate. thisis Group 2 thisis Group 1 Is it possible to use variable shared by different process groups? If not available, are there any other good ways? I would like to periodically monitor the API and give a slight delay according to the API status. When the API is in a bad state, I want to give it time to recover by adjusting the throughput(by ControlRate).
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry