Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI cluster unstable with large number of processors

Highlighted

NIFI cluster unstable with large number of processors

New Contributor

Hi All,

We are developing a nifi solution to read data from 4 SQL server instances and deliver the same data to 4 different SQL instances replicating peer-2-peer SQL replication. For single table, we are using approx 40 processors. When we have less tables on the cluster, nifi works fine, whereas when we add more tables to the cluster, let's say, 150 tables which make total processor count to approx 6000 it becomes almost impossible to scale up the cluster. That means, the cluster behaves fine with 2/2 nodes but as soon as we add another 3rd or 4th node we see the below error messages and it wait forever and makes the cluster unstable. We are running nifi on AWS ec2-instances with c5 infrastructure. 

 

2019-11-22 07:26:53,739 WARN [main] o.a.nifi.controller.StandardFlowService Fail ed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolExcept ion: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from node-2.stage.mktint.global:9999 due to: java.net.SocketTimeoutExceptio n: Read timed out
2019-11-22 07:26:58,741 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at node-2.stage.mktint.global:9999; will use this address for sending heartbeat messages

 

2 REPLIES 2

Re: NIFI cluster unstable with large number of processors

Cloudera Employee

Hi,

 

could you expand on this?

 

For single table, we are using approx 40 processors. When we have less tables on the cluster, nifi works fine, whereas when we add more tables to the cluster, let's say, 150 tables which make total processor count to approx 6000 it becomes almost impossible to scale up the cluster.

 

What processor are you using to query your sql servers?

 

Have you increased the Max thread count on NIFI UI?

Highlighted

Re: NIFI cluster unstable with large number of processors

New Contributor
Hey, Thanks for the revert. We are using below processors for processing of
single table:-
1. QueryDatabaseTable to read records from table.
2. Split records to split one record per flowfile
3. ConvertAvroToJson
4. EvaluateJSONPath to get data in variables
5. ExexuteSQL to insert the records to other servers.
6. A processor group named as "RETRY" which retries the exexuteSQL if it
fails for 3 times.

In the above methodology, processor count for a single table reaches
approximately 40. And, for 150 tables it reaches 6000.

Also, yes, we have increased max timer driver and event driven thread
counts to 100 and 200. But the issue we see is when we try to add a new
node to the cluster. Two node cluster works perfectly fine but as soon as
we make it a three or a four node cluster. It starts to give the
error message pasted in the case.

Any help would be appreciated.
Don't have an account?
Coming from Hortonworks? Activate your account here