NiFi Nodes getting disconnected frequently

[#1]

I have 5 nodes of NiFi cluster. using 3000 processors. which contains 150+ pipelines(which have some simple use cases like reading the date from RDBMS or from file directory, applying some transformation, and writing it to Kudu table or HDFS. and in some use cases we are making http calls by using InvokeHTTP and JOLTTransformation processor).

we are using PutKudu processor in many pipelines. some of the custom processors also we used(like ExecuteGroovyScript, GetSMPP).

PFB the cluster details :

Each node has 24 GB ram.

2 nodes have 16 core, another 2 nodes have 32 core and the remaining node have 48 cores.

Almost all the times, I am not seeing much files in the queue and active threads are also like less than 50 in each node. but still we find the NiFi UI slow. and each node consuming more than 80% of jvm heap space regularly. CPU utilization are also going beyond 80%

Can someone review my configuration or guide me here to resolve this issue.

[#2]

@Vinay91 ,

Do you see any errors in the nifi-app.log file?

Do you have a subscription with Cloudera Support? This is something the support team could help you quickly resolve.

Cheers,

André

[#3]

no error in nifi-app.log. only I can see the warning message "Memory Pool 'PS Old Gen' has exceeded the configured Threshold of 80%, having used 15.49 GB / 16 GB (96.84%)" for the reporting task.

[#4]

Yes, we have Cloudera subscription. We have contacted to support team, and they have asked us to use 'Maximum Timer Driven Thread Count ' as 64. Earlier we were using this count as 280. we have tried to set it to 160 but it didn't help. So, we have reverted it back to 280 again. 

[#5]

Has this been resolved. We are also facing similar issue. 

[#6]

@Althotta 

NiFi node disconnections are rarely the result of some underlying issue in NiFi code.  Node disconnection are more commonly the result of resource consumption or configurations not being optimal in a NiFi deployment.  

This particular post if two years old.  You should create a new post and provide details around your node disconnection issue and the specific Apache NiFi version you are running to get better assistance.  The NiFi cluster UI is a good place to start. 

The "view details" icon to the far left for each node will provide you with node events which will include disconnect events along with reason. 

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt