hello. I'm having a problem with Nifi, so I'm posting a question.
In my Nifi, an error occurs in one node, and there is a problem that the node is disconnected in the cluster
A total of 6 nodes were configured as a cluster with 3 nodes in different datacenters.
The current configuration is collecting network logs through ListenSyslog, ParseCEF, and then sending them to graylog.
It receives an average of 721MB per 5 minutes, but there is a problem that the cluster becomes Disconnected with a Queue Full error occurring on some nodes.
The error message is below.
Communication Failed with Remote Address[null]: java.lang.illegalStateException: Queue full
It says Queue full, but I don't know what kind of queue the error message points to and how to solve it.
My nifi.properties configuration are below.
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.queue.backpressure.count=800000
nifi.queue.backpressure.size=10 GB
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.queue.swap.threshold=100000
nifi.provenance.repository.index.shard.size=8 GB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Volatile Status History Repository Properties
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
The servers have specifications of 48 cores (HyperThread enable) and 256GB Memory.
please help me thank you.😭
