@rahul_ars  You can use the "UpdateCounter" [1] processor to create counters and update the count on them (up and down).  The "Counter Name" property accepts NiFi EL, so you can use attribute son FlowFiles to dynamically update a counter uniquely by FlowFile traversing same dataflow path.  This will allow you to consolidate within your dataflow(s) to reduce the number of processors needed.  Fewer processors can lead to better performance through reduced resource utilization. The "Counters" UI is found under the NiFi Global menu in the upper right corner of the NiFi UI. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateCounter/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thanks, Matt     ... View more

@techNerd  I don't know anything about the Wildfly endpoint service, so I can only assume your URL is correct.  Using the IP as you have should not be an issue as long as the NiFi host can reach that network address. What you are seeing is a timeout which indicates the endpoint you are trying to reach did not respond to you post request.  There could be numerous reasons for this such as network issues, incorrect or missing headers on post request, bad endpoint URL, too many concurrent connections to the endpoint at time of this request, failed authentication, using http for a https endpoint, bad or no client authentication, etc... I would suggest you monitor the logs on Wildfly when the post request is made by NiFi to see if: 1. Wildfly acknowledges receiving the request. 2. Wildfly does not throw and exception about the request. Since the endpoint URL you shared had multipart in it, I assumed that perhaps it only accepts multi-part form data and thus may be expecting the proper header fo this type of data.  And if it is multi-part form data, you are going to have better success using the postHTTP processor instead of the InvokeHTTP processor. Hope this helps you, Matt ... View more

@Kilynn  Can you provide more detail around "nodes randomly restarting"? What do you see in the nifi-app.log and nifi-bootstrap.log when the node(s) restart? Do you see anything in /var/log/messages?  Maybe OOM Killer? I see Max heap shows 330 GB configured.  With 12 nodes that is ~27GB set per node.   Any particular reason why you set heap so high for your NiFi nodes? Larger heaps mean longer stop-the-world GC pauses. What version of Java are you using? Thanks, Matt ... View more

@techNerd  I am not sure how large your multi-part form data is that your are trying to post to you endpoint is, but your timeout values are very aggressive at 5 sec and 15 sec.   Have you tried sending these values to more common 30 sec? Have you tried reaching this endpoint from command line on the host where NiFi is running? How is the rest of yoru processor configured? Try using the PostHTTP [1] processor instead. Make sure you set the "Content-Type" configuration property to match the content that is actually being sent. For example: Content-Type = multipart/form-data; boundary=<boundary id for multipart content> Replace boundary sting with whatever your source content uses in its multipart form data. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PostHTTP/index.html If you found this helped with yoru query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

@dieden9    NiFi provides a number of Kafka processors based off the Kafka Client they are using.  The original ConsumeKafka processor (no number) used the old Kafka 0.8 client.  The 0.8 client processor does not offer the ability to specify a regex for the topic names.  You should be using the Kafka client version processors that match the Kafka server version you are consuming from. From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s). The pattern is a java regular expression.   [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_1_0/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

@Nickanor  It would be interesting to see a verbose listing of your NiFi  logs directory once it has well exceeded 50 GB archived log files. What you have configured will retain 30 hours of log data.  With each of those 30 hours you may have 1 or more incremental log files (each at 100 MB except for last one each hour). On NiFi restart do you see that the following is cleaning up the archive directory of files older than 30 hours: <cleanHistoryOnStart>true</cleanHistoryOnStart>   I would be inspecting the nifi-app.log to see if you encounter any exceptions around logback or if you see any OutOfMemory (OOM) or no more files (file limits) exceptions that may explain the behavior. Hope this helps, Matt ... View more

@kkau  The ERROR you see is because the PrometheusRecordSink controller services when started binds to the configured port as a listener which allows Prometheus server to connect to that port for the purpose fo scraping data from NiFi. On a single host, only one service can bind to a port.  So any subsequent component on that server that attempts to bind to that same port will fail with "address already in use".   I don't know much of anything about Prometheus server, but if you can configure Prometheus to scrape from a unique port on each NiFi instance then you should be able use the variable registry to define a unique port number per host.  By using NiFi Expression Language (EL) you would still have matching flow.xmls on all your NiFi nodes.  You would simply have a unique variable registry file configured for each instance in that instance's nifi.properties file.   So for your nodes you would have three files: <path to file>/node1-variables <path to file>/node2-variables <path to file>/node3-variables In inside each of those files you would have the same property but each with unique value PrometheusPort=9092  (written to node1-variables file) PrometheusPort=9093  (written to node2-variables file) PrometheusPort=9094  (written to node3-variables file) Then when you start this reporting task, each node binds to a different port number.  Then you have your Prometheus server scrape from all three unique endpoints. But I do agree with @TimothySpann that having multiple NiFi hosts run on the same server is not a recommended setup. If you found that any of the provided solutions helped with your query, please take a moment to login and click accept on the solution(s). Thanks, Matt ... View more