LookupRecord queue processor suddenly turns full

Question

Hello,

I have a Nifi Flow working to ingest logs messages and send it up to ElasticSearch. Everything worked fine for one month but suddenly some issues appeared without changes about size and volumetry of incoming logs messages:

- I have a lookupRecord processor to enrich data and the queue become full for it so i guess this processor is the problem but it never had this behavior before

- Nifi tell me it's unable to write flowfile content to content repo due to archive file size constraint

- The input processor of the flow is a listenUDP and warn me that the internal queue is full

I tried to reboot everything, obsioulsy, and also to increase JVM memory settings to Xmx12g to be sure it's not a lack of memory.

I saw that the archive cleanup issue could be a bug from Nifi but I'm actually on 1.17 and even tried to move to 1.20 and everything is same.

Any idea of what could cause those problems ?

Thanks

steven-matison · Accepted Answer

@Pierro6AS

First thing you should do is increase the size of the message queue. The default size is quite low (10,000 records and 1gb). It is possible to see this error if the flowfiles have been in the queue for too long. It is also possible to see this error if the file system has other usage outside of nifi. For best performance nifi's backing folder structure (content/flowfile repository) should be dedicated disks that are larger than the demand of the flow (especially during heavy unexpected volume).

You can find more about this in these posts:

https://community.cloudera.com/t5/Support-Questions/Unable-to-write-flowfile-content-to-content-repository/td-p/346984

https://community.cloudera.com/t5/Support-Questions/Problem-with-Merge-Content-Processor-after-switch-to-v-1-16/m-p/346096/highlight/true#M234750

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.