@Leo3103  This exception might be caused by a missing nar file in your MiNiFi. MiNiFi does not include all the NiFi nars. https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/minifi-java-agent-quick-start.md A quick look through your flow definition I saw you were using the ExecuteSQL processor. While ExecuteSQL is part of one of the included nars, it does require another nar that is not included (this is noted in the above linked quick start guide). You'll need to add the "nifi-dbcp-service-nar" from your NiFi distribution to your MiNiFi's lib directory. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

I think you issue may be with using the SingleUserAuthorizer and Single user login provider.  These out of the box providers were built so that NiFi could be HTTPS enabled securely out of the box.  They are not designed to support clustering, nor are they suitable for production NiFi. You'll want to configure your NiFi cluster to use a production ready authorizer (managed authorizer) and user authentication method other then single user so you can have granular access controls per user/team.  Most common is the ldap-provider. The documentation provides examples for authorizer.xml setup: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-ldap-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referencing-user-dn ETC.... You cluster is most likely not forming completely due to node to node authentication and authorization issue resulting from using the single user authorizer.   In a NiFi cluster the node identities (derived from clientAuth certificates in the Mutual TLS exchange) need to be authorized against some NiFi policies like "Proxy user requests". Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@hegdemahendra  The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.   What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Rohit1997jio  You have topic A with your source messages.  You have two consumer groups each pulling all the messages from Topic A.  While both these dataflows consume all the same messages, each may fail on a different messages with the InvokeHTTP execution.   You are then writing the FlowFiles that failed invokeHTTP to another topic R which both consumer groups can consume from. So both consumer groups will get a copy of any message written to the topic.  Your dataflow is working exactly as designed.  You must keep your retry logic independent of one another.   I also don't understand the overhead of ingesting the same messages twice in your NiFi. Why not have have a single ConsumeKafka ingesting the messages from the topic and then routing the success relationship from the ConsumeKafka twice (once to InvokeHTTP A and once to InvokeHTTP B)?   Why publish failed or retry FlowFile messages to an external topic R just so they can be consumed back in to your NiFi?  It would be more efficient to just keep them in NiFi and create a retry loop on each InvokeHTTP.  NiFi even offers retry handling directly on the relationships with in the processor configuration. If you must write the message out to just one topic R, you'll need to append something to the message that indicates what InvokeHTTP (A or B) failure or retry resulted in it being written to Topic R.  Then have a single Retry dataflow that consume from Topic R, extracts that A or B identifier from message so that it can be routed to the correct invokeHTTP.  Just seems like a lot of unnecessary overhead. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Chetan_mn  Details are very beneficial in getting assistance in the community. Why different NiFi instance would produce two different FlowFile attribute contents, makes no sense to me.  1. Are these two instance of NiFi just  2 nodes in the same NiFi  cluster? 2. What version(s) of NiFi are being used? 3. What processor is being used to your http requests?  I am assuming the HandleHttpRequest processor. 4. How is the HTTP processor configured? 5. Can you share a sample http request? 6. What is the source generating the request?  Is it an automated process? Thank you, Matt ... View more

@AlokKumar  The idea is that after renaming your FlowFile, you use the putFile processor to write that file to your archive directories on each of the NiFi instances.  That ends that dataflow. You then start a new dataflow that consists of only the GetFile processor with its success relationship terminated and the minimum File Age property set to "30 days".  Start this processor and it will continuously check the target archive directory for any files where the last modified timestamp has exceeded 30 days.  Those files older then 30 days will be consumed and removed from archive assuming property "Keep Source File"  is set to "false".  Since the success "relationship" is set to auto-terminate, The FlowFile produced is terminated. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@P2as  Zookeeper needs to have more then 1 node in order to have quorum. A single ZK instance will not work.   The next question is how is client/user authentication and authorization setup? While i dod not see any untrusted proxy exception or SSL exceptions in what you shared, I wonder if you are encountering a mutualTLS issue between your two nodes resulting in your connection exception. When you try to access a NiFi node's URL, your request is replicated by the elected cluster coordinator to all nodes in the cluster and those nodes need to respond with what access the authenticated user is authorized for on each node.  It is this replication request that is failing. You may need to dig a bit deeper in to your logs and configurations making sure that the NiFi instances successfully bound to those ports and those ports are not being blocked. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more