Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flow file repository failed to update Errors - Done all tweaks , Need some help

Solved Go to solution
Highlighted

Flow file repository failed to update Errors - Done all tweaks , Need some help

Expert Contributor

Hi All,

We have just productionized our Nifi flows which is a real time streaming application . We have a cluster of 2 Nodes and each node has 6 processors and a high memory of 28 GB. On the Nifi side the memory has been set to 24 GB. Moreover , we tend to see a queue size reaching to around 90,000 or more. For this I have set the property nifi.queue.swap.threshold to 90000 . We also have a disk space of 200 GB.

Despite having such a powerful configuration , we still get flow file repository failed to update. I understand that initially the JVM memory is uitilized for the flow file processing and if that gets overused , it spills over to disk space . Our disk space is also pretty high at 200 GB however we still keep on seeing this error.


Any pointers or suggestion in this space will be of great help.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Flow file repository failed to update Errors - Done all tweaks , Need some help

Expert Contributor

For now we set the nifi.queue.swap.threshold to 150000 and this seems to have done the trick. Loads are performing well and there is no build up of the queue. thanks all

1 REPLY 1

Re: Flow file repository failed to update Errors - Done all tweaks , Need some help

Expert Contributor

For now we set the nifi.queue.swap.threshold to 150000 and this seems to have done the trick. Loads are performing well and there is no build up of the queue. thanks all