Hey, Thanks for the revert. We are using below processors for processing of single table:- 1. QueryDatabaseTable to read records from table. 2. Split records to split one record per flowfile 3. ConvertAvroToJson 4. EvaluateJSONPath to get data in variables 5. ExexuteSQL to insert the records to other servers. 6. A processor group named as "RETRY" which retries the exexuteSQL if it fails for 3 times. In the above methodology, processor count for a single table reaches approximately 40. And, for 150 tables it reaches 6000. Also, yes, we have increased max timer driver and event driven thread counts to 100 and 200. But the issue we see is when we try to add a new node to the cluster. Two node cluster works perfectly fine but as soon as we make it a three or a four node cluster. It starts to give the error message pasted in the case. Any help would be appreciated.
... View more
We are developing a nifi solution to read data from 4 SQL server instances and deliver the same data to 4 different SQL instances replicating peer-2-peer SQL replication. For single table, we are using approx 40 processors. When we have less tables on the cluster, nifi works fine, whereas when we add more tables to the cluster, let's say, 150 tables which make total processor count to approx 6000 it becomes almost impossible to scale up the cluster. That means, the cluster behaves fine with 2/2 nodes but as soon as we add another 3rd or 4th node we see the below error messages and it wait forever and makes the cluster unstable. We are running nifi on AWS ec2-instances with c5 infrastructure.
2019-11-22 07:26:53,739 WARN [main] o.a.nifi.controller.StandardFlowService Fail ed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolExcept ion: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from node-2.stage.mktint.global:9999 due to: java.net.SocketTimeoutExceptio n: Read timed out 2019-11-22 07:26:58,741 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at node-2.stage.mktint.global:9999; will use this address for sending heartbeat messages
... View more