@rahul_ars  You can use the "UpdateCounter" [1] processor to create counters and update the count on them (up and down).  The "Counter Name" property accepts NiFi EL, so you can use attribute son FlowFiles to dynamically update a counter uniquely by FlowFile traversing same dataflow path.  This will allow you to consolidate within your dataflow(s) to reduce the number of processors needed.  Fewer processors can lead to better performance through reduced resource utilization. The "Counters" UI is found under the NiFi Global menu in the upper right corner of the NiFi UI. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateCounter/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thanks, Matt     ... View more

@techNerd  I don't know anything about the Wildfly endpoint service, so I can only assume your URL is correct.  Using the IP as you have should not be an issue as long as the NiFi host can reach that network address. What you are seeing is a timeout which indicates the endpoint you are trying to reach did not respond to you post request.  There could be numerous reasons for this such as network issues, incorrect or missing headers on post request, bad endpoint URL, too many concurrent connections to the endpoint at time of this request, failed authentication, using http for a https endpoint, bad or no client authentication, etc... I would suggest you monitor the logs on Wildfly when the post request is made by NiFi to see if: 1. Wildfly acknowledges receiving the request. 2. Wildfly does not throw and exception about the request. Since the endpoint URL you shared had multipart in it, I assumed that perhaps it only accepts multi-part form data and thus may be expecting the proper header fo this type of data.  And if it is multi-part form data, you are going to have better success using the postHTTP processor instead of the InvokeHTTP processor. Hope this helps you, Matt ... View more

Thanks a lot ... View more

Thanks @MattWho    Sure, I will consider your suggestion to not running multiple nifi on the same machine.   I tried the variable registry approach as well but the problem is the same with that as well that we can not use the EL parameter. ... View more

@dieden9    NiFi provides a number of Kafka processors based off the Kafka Client they are using.  The original ConsumeKafka processor (no number) used the old Kafka 0.8 client.  The 0.8 client processor does not offer the ability to specify a regex for the topic names.  You should be using the Kafka client version processors that match the Kafka server version you are consuming from. From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s). The pattern is a java regular expression.   [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_1_0/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

Thanks, it's help me a lot. ... View more

@Nickanor  It would be interesting to see a verbose listing of your NiFi  logs directory once it has well exceeded 50 GB archived log files. What you have configured will retain 30 hours of log data.  With each of those 30 hours you may have 1 or more incremental log files (each at 100 MB except for last one each hour). On NiFi restart do you see that the following is cleaning up the archive directory of files older than 30 hours: <cleanHistoryOnStart>true</cleanHistoryOnStart>   I would be inspecting the nifi-app.log to see if you encounter any exceptions around logback or if you see any OutOfMemory (OOM) or no more files (file limits) exceptions that may explain the behavior. Hope this helps, Matt ... View more

Thank you.   That is the case, there is a lot going on and we are trying to determine the sizing needed for new processes.    We did a POC with the need to load about 6 million records.   When it came time to implement in Test the cluster isn't enough to handle the load since the data had grown to 173 million.   Can't wait to see what is in Prod.   🙂   Thank you for the explanation about the bootstrap and child as well as the suggestions!   Much appreciated.    ... View more

@MattWho    Thanks for the time available for my problem, the "ExtractText" worked perfectly for my case and the data was saved in my database.   Grateful. ... View more