Yes Paras. It is clear now. Thanks. However, any inputs on the following is highly appreciated. Currently, I have Nifi running on an edge node that has 4 cores. Say I have 20 incoming flow files and I give concurrent tasks as 10 for ExecuteStreamCommand processor, does it mean I get only concurrent execution or both concurrent and parallel execution? ... View more

Thanks for your reply Paras. Currently I designed the flow as SelectHiveQL(reading as csv instead of default avro)->SplitText(By Line) ->ExtractText (Here assigning content of the split files to an attribute). This is good so far. Every value of my query result is associated with a flow file attribute. And, I hope this is what you were also mentioning; but in a different way. Now the question is about the ExecuteStreamProcessor where I pass the flow file attribute to command arguments. So, here could you please clarify if one task handles one spark-submit command with attribute from one flowfile taken at a time? Is my understanding correct? I remember reading somewhere that one task in Nifi can process multiple flow files at a time. So, wanted to understand how the flowfiles are handled by Nifi processor tasks. Regards, John ... View more

Hi Wyner, Yes, after i disabled and enabled the MapCahce service configuration, it was working fine. Thanks for that. ... View more

I am using file name as my signal identifier. Filenames before notify and wait are all renamed to a specific value and the count of these signals is what I am checking. ... View more

Thanks for your reply, Wyner. Here is the screen shot of the flow that I simulated from the actual ones that we have. Also, the configurations of wait and notify that I am using are shown here: 1. Generate flow file generates a flow file every 5 seconds 2. Flow files are renamed to 'release_signal' (as this is what I am using as release signal identifier in wait and notify processors) 3. When there are 5 such signals, I want wait processor to push all the 5 flow files to the downstream success relation. Schedule on wait processor is 10 sec Given the above scenario, I expect the success relation from wait processor should get flow files in the steps 5.. (like 5,10,15 ..) But I do not see something like that. You may see it in the screen shot too. There are 13 files in total that are pushed to success relation from wait. What I observed is, sometimes 5 flow files are pushed at once. But sometimes even one flow file is pushed to success from wait which I don't understand why that happens. Since I could not get this working as expected, I switched to using "Merge Content" and it works for the use case that I have at hand. Flow: Update Attribute: Notify: Wait: ... View more

Hi Wyner, wait-processor.jpeg You can see that am waiting for 10 signals here. ... View more

Hi, Could you please let me know how the resetting of target signal count in wait processor work? Refer my question: https://community.hortonworks.com/questions/138762/reset-of-target-signal-count-in-wait-processor.html Thanks, John ... View more

Hi, Could you please let me know how the resetting of target signal count in wait processor work? Refer my question: https://community.hortonworks.com/questions/138762/reset-of-target-signal-count-in-wait-processor.html Thanks, John ... View more