Member since
11-29-2022
2
Posts
0
Kudos Received
0
Solutions
01-19-2023
07:21 AM
hello. I'm having a problem with Nifi, so I'm posting a question. In my Nifi, an error occurs in one node, and there is a problem that the node is disconnected in the cluster A total of 6 nodes were configured as a cluster with 3 nodes in different datacenters. The current configuration is collecting network logs through ListenSyslog, ParseCEF, and then sending them to graylog. It receives an average of 721MB per 5 minutes, but there is a problem that the cluster becomes Disconnected with a Queue Full error occurring on some nodes. The error message is below. Communication Failed with Remote Address[null]: java.lang.illegalStateException: Queue full It says Queue full, but I don't know what kind of queue the error message points to and how to solve it. My nifi.properties configuration are below. nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=800000
nifi.queue.backpressure.size=10 GB
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.queue.swap.threshold=100000
nifi.provenance.repository.index.shard.size=8 GB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min The servers have specifications of 48 cores (HyperThread enable) and 256GB Memory. please help me thank you.😭
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
12-06-2022
02:35 PM
@dreaminz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks
... View more