@AlokKumar  There are a lot of steps involved with setting up a NiFi cluster.  Without knowing the details of your standalone NiFi configuration, giving specific guidance on what it would take to convert your standalone to a cluster would be challenging and lengthy.  I'd recommend reading through the https://nifi.apache.org/nifi-docs/administration-guide.html and setting up a separate NiFi cluster to understand all the configurations that need to be made to be successful before implementing this on your production env.  https://nifi.apache.org/nifi-docs/administration-guide.html#clustering  https://nifi.apache.org/nifi-docs/administration-guide.html#basic-cluster-setup Basic requirements: 1. NiFi cluster requires Zookeeper (ZK): min 3 ZK nodes..  ZK must have quorum to work correctly.  ZK is utilize to facilitate election of NiFi cluster node roles cluster coordinator node (handle request replication to all nodes in NiFi cluster) and Primary node (only node that will schedule "primary node" only configured canvas components).  NiFi offers the ability to start an embedded ZK, but keep in mind that if NiFi is stopped on a node it also stops that NiFi's ZK.  ZK is also used to store shared cluster state for processors that need to share state across all nodes.  https://nifi.apache.org/nifi-docs/administration-guide.html#embedded_zookeeper 2. You'll need to configure all NiFi nodes the same (configuration files should match except node specific values).  3. You'll want to use a more robust configurable authorizer which means also using some form of user authentication besides default single-user enabled out-of-the-box for standalone NiFi evaluation.   ldap-provider, kerberos-provider, or SSL certificates are most commonly used, but other options can be found in admin guide as well. https://nifi.apache.org/nifi-docs/administration-guide.html#user_authentication https://nifi.apache.org/nifi-docs/administration-guide.html#multi-tenant-authorization https://nifi.apache.org/nifi-docs/administration-guide.html#config-users-access-policies Hope this help get you started on your clustered NiFi journey. I encourage you to create more community questions as you explore this option should you run into complications. Thanks, Matt   ... View more

@zzzz77  Still not clear on the entire workflow you have going on between your Windows NiFi and 3 node NiFi Cluster. 1. "where the acknowledgement files get stuck between the cluster and the client machine" - not clear in where exactly this means.  Queued in some connection with the NiFi canvas between what two components?  Can you share screenshots of your dataflow setup showing where they get stuck?  Is back pressure being applied to any of the connections? 2. "We also observed files would "disappear" when we had 2 downstream sites connected to the same output port" - please elaborate here.  It is common to have multiple RPGs connected to remote ports (if the RPG is on a 3 nodeNiFi cluster, then you have 3 RPGs attempting to pull data from the target Remote output port).  No different then if you had three non clustered standalone NiFi instances all connected to same Remote OutPut Port (all are trying to connect constantly and get FlowFiles from the port).  I have not observed FlowFiles "disappear" with RPG.  Did you use NiFi's built in Provenance to search for a "disappeared" FlowFile using the lineage? I would recommend not building your flow around "Remote output ports".   There is no good load distribution happening with "Remote Output ports".  The "Remote Process Group" (RPG) is the client in all Site-To-Site transfers.   When you add and configure a RPG, it will connect to the target the first Target URL configured and fetch the Site-to-Site (S2S) details for that target cluster. While you can configure multiple URLs in a comma separated list in the RPG, it only attempts the next configured URL if first is not reachable.   Those S2S details contain info from the target NiFi instance/cluster (to include but not limited to: num nodes in cluster, support protocol, s2s ports, individual node load, etc). These details allow the RPG to create a distribution plan.  lets assume target is a 3 node cluster and node one has 10,000 queued FlowFiles, node 2 has 5,000, and node 3 has 5,000 queued FlowFiles.  Since node 1 has a higher load, the RPG would try to make sure nodes 2 and 3 got more FlowFile sent to them.  So distribution might result in transfers don in order like this: (node 1, node 2, node 3, node 2, node 3, repeat from beginning).  So you will notice with each iteration it send twice to nodes 2 and 3 and only once to node 1.  The RPG has no round robin configuration.   But even with above, there is no guarantee of any even or close to even load distribution. Under continuous dataflow load, you will see pretty good distribution, but the fewer FlowFiles the less distributed it can become. Below Article covers the settings that can help improve the distribution of FlowFile across nodes via RPG. https://community.cloudera.com/t5/Community-Articles/How-to-achieve-better-load-balancing-using-NiFi-s-Site-To/ta-p/246279 Form above article you learn what configurations exist when "sending" FlowFiles to a Remote Input Port.  But Output Ports are different.  The RPG (client) is still connected to the "output port" of your target NiFi instance or cluster.  Lets say the RPG is on you 3 node cluster, so that means that each node in the 3 node cluster has its own copy of the RPG executing.  So each node polls the output port to fetch FlowFiles.  There are no controls to limit how much data 1 node's RPG may pull.  It simply connects and pulls everything currently based on output port config settings on RPG, but there is no distribution model since it is a pull.  So as soon as it finishes it will attempt again.   So you have less control over over distribution when using Remote Output Ports. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  I agree with @vafs  that sharing the completed ERROR and stack trace is always gong to be most helpful in your community questions.  Those full stack traces will have classes like "net.schmizz.sshj.transport" that you could try putting in to DEBUG within the NiFi logback to see what additional logging that class (not a NiFi library, but used by NiFi) may provide.  Matt ... View more

@zzzz77  Sorry to hear you are having challenges with your dataflow. Can you clarify "site-to-site connection queue after the cluster" as this is not very clear. Are FlowFiles always stuck in same connection queue?  Can you share a screenshot? Is this a connection to a Remote Process Group, Remote Output Port?  Of a connection between a Remote Input Port and some other NiFi component (what is this component and how is it been configured? What is the configuration settings of the connection with the "stuck/delayed" FlowFiles? Thank you, Matt ... View more

@oka  Perhaps others in the community may hav additional suggestions here, but since the "-" is not a valid character in JMS, you would need to use a AMQP processor to support these headers. As mentioned before there is a https://issues.apache.org/jira/browse/NIFI-14670 jira for adding AMQP 1.0 support to ConsumeAMQP processor, but it is still open and unassigned. Now that jira points to using the Qpid JMS Client in ConsumeJMS and as you experienced it works but still has limitations.  Those limitations impact these specific properties with the "-" in the name.  I would suggest adding your experience with trying to use Qpid AMQP in the above jira and what impacts it has on the two headers you require to maybe push the Apache community to adding AMQP 1.0 support to the AMQP specific processors. Additionally, there is this jira (https://issues.apache.org/jira/browse/QPID-4992), where an individual expressed some success preserving the content type header by using ActiveMQ JMS API instead of the Qpid AMQP JMS API.  So you may want to give this jira a read and maybe try this for yourself.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more