Below is a blog post for a flow i made using the nifi api, which will accept a template (XML), load the template, get template, then follow chain of api calls to get the appropriate flow definiton file (JSON). https://cldr-steven-matison.github.io/blog/dataflow-xml-to-json/   The concepts you need to understand are demonstrated with invokeHttp to execute the sample calls. Check the nifi api docs.  You can do these calls with curl, etc., but i chose to do them in nifi.   Be mindful that to start using the api from outside of nifi,  you need to be following the initial api call for an access token to be used with the rest of the calls.  ... View more

Thanks @MattWho , Maybe we tried to skip versions and created this problem. But we could solve it. @gourexe  I think the solution that we have applied at that moment, was to migrate the flow to 1.16.3 in a fresh installation. Reconfigure all the secrets and then move on the versions. The lesson that we learned is that we should have a process to test and apply new versions from Nifi, as soon as possible. 🙂 ... View more

ok thank you ... View more

I think you issue may be with using the SingleUserAuthorizer and Single user login provider.  These out of the box providers were built so that NiFi could be HTTPS enabled securely out of the box.  They are not designed to support clustering, nor are they suitable for production NiFi. You'll want to configure your NiFi cluster to use a production ready authorizer (managed authorizer) and user authentication method other then single user so you can have granular access controls per user/team.  Most common is the ldap-provider. The documentation provides examples for authorizer.xml setup: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-ldap-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referencing-user-dn ETC.... You cluster is most likely not forming completely due to node to node authentication and authorization issue resulting from using the single user authorizer.   In a NiFi cluster the node identities (derived from clientAuth certificates in the Mutual TLS exchange) need to be authorized against some NiFi policies like "Proxy user requests". Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@hegdemahendra  The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.   What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

Thanks for the detailed reply. I added a new value in the nifi.properties and it worked as my installation was brand new, but did not know that the exact value needs to be copied over from the previous installation if we are configuring an old installation. Thanks again. ... View more

@Salmidin  Please create a new community question with the details of your issue. It seem unrelated to this thread issue.  The "Policies" missing from the global menu indicates your NiFi is either using Single User Authorizer (default out of box) or is not setup to be secure. NiFi needs to be configured with a production ready authorizer for the "Polices" and "Users" options to be visible in the NiFi Global menu. Fell free to ping me @MattWho in your new community question. Thanks, Matt ... View more

@NagendraKumar  The image you shared indicates that your PublishKafka processor is producing a bulletin.  What is the nature of the exception being reported in that bulletin? I also see what appears to be only one connection exiting your PublishKafka processors.  The publishKafka processor has multiple relationships. Are you auto-terminating the failure" relationship?  If so, i never recommend doing that. ----------- Now when it comes to monitoring queue sizes/thresholds, you could use the ControllerStatusReportingTask NiFi reporting task to output these stats for all connections to a dedicated logs (see Additional Details... for how to setup dedicated logs via NiFi's logback.xml).   You can then create a NiFi dataflow that tails the dedicated connection log and parses the ingested log entries for connections where thresholds exceed your 80% threshold and route those to a putEmail processor for yoru notification needs. (preferred) Another option here is to use the SiteToSiteStatusReportingTask to report specifically on NiFi connections and feed that data in to a dataflow that parses for threshold in excess of 80% and route those to a putEmail processor.  This method has less overhead as it does not write to a NiFi log, require tailing logs, can be setup to report on connections, and reports in a more structured format (see Additional Details... ). ---------- From your image I can also see your PublishKafka processor reporting 3 actively running threads.  You mentioned the processor becomes hung?  Have you analyzed a series of thread dumps to identify where it is getting hung?   NiFi also offers a way to monitor for long running tasks: Runtime Monitoring Properties You could use this in conjunction with the SiteToSiteBulletinReportingTask to construct a dataflow that could send an email alert when task are detected on processor that have been running in excess of the configured threshold.  This runtime monitoring does have an impact on your overall NiFi performance due to the overhead needed to run it.  So if you find it impacts your throughput performance negatively, you'll need to stop using it. __________ Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@wasabipeas @Adhitya  In the thrown exception it reports which node has the mismatched revision. Is that node the currently elected cluster coordinator? Have you tried on just that reported node: 1. Stopping NiFi 2. Remove or rename both the flow.xml.gz and flow.json gz files (only deleting one will not work) 3. Restart that NiFi node.  It will inherit the flow from the cluster coordinator when it joins. If this node was the elected cluster coordinator, when you shut it down another node will assume the cluster coordinator role. ---- Another option is to disconnect the node reporting the mismatch in revision.  Then from the same cluster UI used to disconnect that node, select to drop/delete it from cluster.  Your cluster will now report one less node.  See if you can then move the process group or if it reports another node with mismatched revision? NOTE: Deleting/Dropping a node from cluster using the Cluster UI does nothing to that node.  If you restart that node that was deleted/dropped it will rejoin the cluster again.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more