@domR    I see no issues with using a publish and consume design for triggering follow on flows. That provide a slightly more robust setup than using the postHTTP to ListenHTTP example I provided since any NiFi node can consume the trigger file. When using Kafka you will want to make sure you have the same number of partitions as you have consumer in the same consumer group.  When you add a consumeKafka processor to the NiFi canvas it is really being added to every node in your cluster and is configured the same on every node. Let's assume you have a 3 node NiFi cluster, that means you have at a minimum 3 consumers in that consumer group so your Kafka topic should have 3 partitions; otherwise, you may see a lot of rebalancing happen.  To improve performance even more you can increase the concurrent task on your consumeKafka processor.  With a 3 node NiFi cluster and a consumeKafka configured with 2 concurrent tasks, you now have 6 total consumers in the consumer group; therefore, your Kafka topic should have 6 partitions.   If a NiFi node goes down Kafka will assign multiple partitions to the same consumer, so no need to worry about messages not being consumed. Hope this information was useful in finding a solution, Matt ... View more

@Umakanth    The API is exposed out of the box, it is not something you need to enable. Every action you take while performing actions within the UI makes a call to the NiFi rest-api. When learning how to use the rest-api calls, you may find using the developer tools in your browser helpful. Open the developer tools  while you are accessing your NiFi UI.  Then perform some action and you will see those requests being made by your browser to NiFi.  In the below example (Using Chrome browser developer tools), I opened NiFi's summary UI from the global menu in the upper right corner of the UI: You'll notice that several requests were made.  I can write click on any one of those request and select "Copy as cURL" to copy the full request to the system clipboard. I can then paste the request in a terminal window and see what the rest-api call returns.  You will notice that the curl command that is copied will have numerous additional headers (-H) that are not always necessary depending on the rest-api endpoint being used.  Example: curl 'http://<nifi-hostname>:<nifi-port>/nifi-api/flow/process-groups/root/status?recursive=true' Of course you will need to parse the rest-api returns yourself to extract in many cases the specific details/stats you want to monitor. Hope this helps, Matt ... View more

Mattwho,             Thanks for your comments. After reading your mail, I spent a lot of time thinking. I saw that 8 threads were created, and there was no performance improvement because all the threads were doing the same thing and executing the script on all files that were unpacked. Later your comment that the threads actually operate on the flowfiles, I changed the code so that it accepts the flow files as  one of the input and processes using multiple threads.  This improved the performance by 30%. The time taken dropped from 13 mins to 3-4 mins.    So many thanks for your comments, I now understand how to use concurrent tasks. ... View more

@maryem    Any action you can do through the NiFi UI, you can also do via interacting directly with the NiFi rest-api. This will not animate the action of actually dragging and dropping a processor on the canvas, but you can make a rest-api call that would add new processor of type ABC at coordinates x,y on the canvas.   NiFi's rest-api documentation can be found here: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html Some users find it easier to learn the rest-api call through examples.  If you open the developer tools in your browser, you can perform the action via the UI and see the rest-api call that was made.  Most browser developer tools even let you save the rest-api call as a curl command that you could then execute yourself via command line. Matt ... View more

@nishank_paras    You can use the invokeHTTP processor to fetch your file.  Here is an example: The above example fetches the Apache nifi-toolkit-1.11.3-bin.tar.gz file. You can then construct a dataflow using other processors to manipulate as you want or simply just connect to another invokeHTTP processor that instead of "GET" uses "PUT" to put your file "nn.csv" at the new http endpoint.   Hope this helps you, Matt   ... View more