The full info from the app log is: 2021-05-17 14:57:12,166 INFO org.apache.nifi.web.api.ParameterContextResource: Stopping 0 Processors in order to update Parameter Context 2021-05-17 14:57:12,173 ERROR org.apache.nifi.web.api.ParameterContextResource: Failed to update Parameter Context org.apache.nifi.web.util.LifecycleManagementException: Failed while waiting for components to transition to state of STOPPED at org.apache.nifi.web.util.ClusterReplicationComponentLifecycle.scheduleComponents(ClusterReplicationComponentLifecycle.java:133) at org.apache.nifi.web.api.ParameterContextResource.stopProcessors(ParameterContextResource.java:960) at org.apache.nifi.web.api.ParameterContextResource.updateParameterContext(ParameterContextResource.java:852) at org.apache.nifi.web.api.ParameterContextResource.lambda$submitUpdateRequest$17(ParameterContextResource.java:804) at org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   ... View more

@MattWho , thanks for your response.  Before I had heard from you I had checked /var/log/messages and searched for OOM messages without success.  However, there was no indications of nifi dying in nifi-app.log so I dug a little farther.  As it turned out I did find the OS killing the child process in NIFI.  I reduced the JVM to 24gb and the OS no longer feels the need to kill it.  So that problem is solved.  As for the JVM, we were attempting to keep as much of our flow queues in memory to reduce disk i/o.  Is there a better strategy? ... View more

@rahul_ars  You can use the "UpdateCounter" [1] processor to create counters and update the count on them (up and down).  The "Counter Name" property accepts NiFi EL, so you can use attribute son FlowFiles to dynamically update a counter uniquely by FlowFile traversing same dataflow path.  This will allow you to consolidate within your dataflow(s) to reduce the number of processors needed.  Fewer processors can lead to better performance through reduced resource utilization. The "Counters" UI is found under the NiFi Global menu in the upper right corner of the NiFi UI. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateCounter/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thanks, Matt     ... View more

@techNerd  I don't know anything about the Wildfly endpoint service, so I can only assume your URL is correct.  Using the IP as you have should not be an issue as long as the NiFi host can reach that network address. What you are seeing is a timeout which indicates the endpoint you are trying to reach did not respond to you post request.  There could be numerous reasons for this such as network issues, incorrect or missing headers on post request, bad endpoint URL, too many concurrent connections to the endpoint at time of this request, failed authentication, using http for a https endpoint, bad or no client authentication, etc... I would suggest you monitor the logs on Wildfly when the post request is made by NiFi to see if: 1. Wildfly acknowledges receiving the request. 2. Wildfly does not throw and exception about the request. Since the endpoint URL you shared had multipart in it, I assumed that perhaps it only accepts multi-part form data and thus may be expecting the proper header fo this type of data.  And if it is multi-part form data, you are going to have better success using the postHTTP processor instead of the InvokeHTTP processor. Hope this helps you, Matt ... View more

@ThangND-2210  Thank you for filing Apache Jira detailing your observations here: https://issues.apache.org/jira/browse/NIFI-8541 I see that Apache NiFi community committers are looking in to your Jira. Thank you, Matt ... View more

@dieden9    NiFi provides a number of Kafka processors based off the Kafka Client they are using.  The original ConsumeKafka processor (no number) used the old Kafka 0.8 client.  The 0.8 client processor does not offer the ability to specify a regex for the topic names.  You should be using the Kafka client version processors that match the Kafka server version you are consuming from. From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s). The pattern is a java regular expression.   [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_1_0/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

@Nickanor  It would be interesting to see a verbose listing of your NiFi  logs directory once it has well exceeded 50 GB archived log files. What you have configured will retain 30 hours of log data.  With each of those 30 hours you may have 1 or more incremental log files (each at 100 MB except for last one each hour). On NiFi restart do you see that the following is cleaning up the archive directory of files older than 30 hours: <cleanHistoryOnStart>true</cleanHistoryOnStart>   I would be inspecting the nifi-app.log to see if you encounter any exceptions around logback or if you see any OutOfMemory (OOM) or no more files (file limits) exceptions that may explain the behavior. Hope this helps, Matt ... View more

@techNerd    Details around your use case would be helpful here.  The putSFTP processor is designed to write FlowFile content to a target SFTP server location.  If you have no content to write, there is nothing for it to do. Thanks, Matt ... View more

@sangee    Details around your use case may be helpful. The SplitText processor will output all FlowFiles produced at the exact same time to the "splits" relationship.  So if you intent is to wait until all splits from a single source FlowFile are produced before processing those splits, this flow is not needed. Pierre has written an excellent blog around using wait and notify processors in a dataflow that does merge and split.  Check it out here: https://pierrevillard.com/2018/06/27/nifi-workflow-monitoring-wait-notify-pattern-with-split-and-merge/   If you found this helpful with your query, please take a moment to login and click accept on this solution. Thanks, Matt ... View more

@syntax_  NiFi does not provide a method for uploading templates in bulk.   But anything you can do via the UI, you can also do via rest-api calls [1] through curl.  So you could script the bulk upload through rest-api external to NiFi's UI. I would strongly discourage bulk uploading templates to NiFi.  Templates should be uploaded as needed, instantiate to the canvas and then template deleted from NiFi.   All uploaded templates even if not instantiated to the canvas become part of the flow.xml loaded in to heap memory.  So keeping a large number of templates uploaded to your NiFi can have a considerable impact on JVM heap memory usage because now you have not only your active flow on the canvas in memory, but also these templates. Additionally, In a NiFi cluster on startup the flow.xml.gz is uncompressed, loaded in to heap memory, and a flow fingerprint created on every node.  These flow fingerprints are then compared to make sure all nodes joining the cluster are running with the same flow.xml.  Since these templates can just add unnecessary size to the flow.xml.gz, this can impact startup times and flow fingerprint comparison time. A better approach is to get your reusable flow migrated into NiFi-Registry [2].  You can have 1 to many NiFi's all connect to a single NiFi-Registry where all your flows exist.  Then all you need to do is drag a process group to the canvas and select "import" which will allow you to select one of these flows from NiFi-Registry and add it to your canvas.  Now you have your flow without the additional impact on heap from having it in NiFi both as a template and as a dataflow on your canvas with easy access to load same flow over and over.  I encourage you to explore NiFi-Registry. [1] http://nifi.apache.org/docs/nifi-docs/rest-api/index.html [2] https://nifi.apache.org/registry.html   If you found this response helped with your query, please take a moment to login in and click accept on this solution. Thanks, Matt ... View more

@dragfly  There are a number of ports that can be set via the nifi.properties file that you would need to make sure are open/reachable.  Some of those you already identified in your question.  The other ports may vary depending on the version of NiFi you are using since new capabilities/features get introduced from time to time. This list includes below properties that would exist in nifi.properties file (depending on NiFi version): Web properties: nifi.web.http.port nifi.web.http.port.forwarding nifi.web.https.port nifi.web.https.port.forwarding Cluster Node properties: nifi.cluster.node.protocol.port nifi.cluster.load.balance.port Site-To-Site properties: nifi.remote.input.socket.port NiFi clusters also have a dependency on zookeeper, so you'll need to make sure your NiFi nodes can reach your ZK hosts over the configured ZK ports configured in your ZK. Then once you start building your dataflows, you may add components to the canvas the introduce additional ports you may need to open.  For example, any processor or controller service that creates a listener for incoming request like ListenHTTP, ListenTCP, DistributedMapCacheServer, etc. For egress the same applies for any external service(s) your NiFi dataflows will need to connect with. If you found that this solution help you, please take a moment to login and click accept on this solution. Thanks, Matt     ... View more

@AnkushKoul    You may consider having two dataflows here because you would need to store this password somewhere that your putSFTP processor dataflow would need have access to. One dataflow that acquires the password from the rest endpoint and another that is actually performs your putSFTP. Flow to get password would consist of a GenerateFlowFile (configure to generate a FlowFIle on each NiFi node using a run schedule to control how often password will be refreshed)--> invoketHTTP --> putDistributedMapCache (Configured to use DistributedMapCacheClient that uses "localhost" as server hostname. Will need to also add a DistributedMapCacheServer Controller service that above client will talk to.  This will make sure each node grabs password and stores it locally on that nodes's DMCServer) Then in your flow using putSFTP, you would replace the invokeHTTP processor a fetchDistributedMapCache processor just before the putSftp processor (suggest looping failure relationship from putSFTP back to fetchDistributedMapCache processor  just in case password changes or there was no password stored when fetch was attempted first time). Keep in mind that flows like this are not idea.  The retrieved password that you are placing in to a FlowFile attribute is stored in plaintext. If you found this solution helpful, please take a moment to login and click accept on this solution. Good luck, Matt ... View more

Hi,   Im also trying same nifi cluster on ecs configuration and facing zombie node issue, node are disconnected  but not able to delete them do you guys get any solutions? ... View more

@naga_satish  If you NiFi is running secured (HTTPS), then every action must be authenticated and authorized and this includes calls made to the NiFi rest API. NiFi by default will always support a client/user certificate for authentication as the first attempted method of authentication.  Using a valid client/user certificate is the most common method used when interacting with NiFi via the rest-api because it does not require the client to acquire a bearer token like other methods such as the ldap-provider.  Also Bearer tokens are only valid on the node from which it was acquired (Can't use token issued by node 1 on node 2, 3, 4, etc...). The best way to to see how these commands are executed is using the developer tools in your browser while executing the actions via the NiFI Ui directly. You can even copy the curl command from the developer tools.  For example, in Chrome browser click on the settings --> more tools --> Developer tools From UI that opens click the "Network" tab. Now you can see the calls being made as you perform them via the UI and can right click on any call and select copy as curl. Now if you still want to use Tokens, here is one example based off using a login provider: To get a token for a user: curl 'https://<nifi-hostname>:<nifi-port>/nifi-api/access/token' --data-raw 'username=<username>&password=<password>' --compressed --insecure The return from the above is your bearer token for the user.  This bearer token is only valid for the duration of the configured expiration in the login-identity-providers.xml file. The following command then can be used to fetch the current state on a processor (<TOKEN> is the string returned from above): curl 'https://<nifi-hostname>:<nifi-port>/nifi-api/processors/<processor-UUID>/state' -H 'Authorization: Bearer <TOKEN>' --compressed --insecure The following command then can be used to clear the state on a processor: curl 'https://<nifi-hostname>:<nifi-port>/nifi-api/processors/<processor-UUID>/state/clear-requests' -X 'POST' -H 'Authorization: Bearer <TOKEN>' --compressed --insecure Since you setup may be totally different in how it is setup to authenticate your users, it is best to use the browser's developer tools to to see the rest-api action in progress to understand the interactions. Hope this helps, Matt ... View more

check disk space   http://apache-nifi-users-list.2361937.n4.nabble.com/Clarifications-on-getting-flowfiles-using-FlowFileFilters-td7333.html     ... View more

@Law    While Jolt transforms are not something NiFi specific and not something I am strong with myself, you may find these links helpful to you: https://intercom.help/godigibee/en/articles/3096940-simple-if-else-with-jolt https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350 Hope this helps, Matt ... View more