@Raj B Not all Errors are equal. I would avoid lumping all failure relationships into the same error handling strategy. Some Errors are no surprise, can be expected to occur on occasion, and may be a one time thing that resolves itself. Lets use your example above.... The putHDFS processor is likely to experience some failure over time do to events outside of NiFi's control. For example, let say a file in the middle of transferring to HDFs when the network connection is lost. NIFi would in turn route that FlowFile to failure. If that failure relationship had been routed back on the putHDFS, it would have likely been successful on the subsequent attempt. A better error handling strategy in this case may be to build a simple error handling flow that can be used when the type of failure might lead to self resolution. So here you see Failed FlowFiles enter at "data", they are then checked for a failure counter and if one does not exist it is created and set to 1. If it exists, it is incremented by 1. The check recount count will continue to pass the file to "retry" until the same file has been seen x number of times. "Retry" would be routed back to the source processor of the failure. after x attempts the counter is reset, an email is sent, and the file is place in some local error directory for manual intervention. https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1&modificationDate=1433271239000&api=v2 The other scenario is where the type of failure is not likely to ever correct itself. Your mergeContent processor is a good example here. If the processor failed to merge some FlowFiles, it is extremely likely to happen again, so there is little benefit in looping this failure relationship back on the processor like we did above. In this case you may want to route this processors failure to a putEmail processor to notify the end user of the failure and where it occurred in the dataflow. The success of the putEmail processor may just feed another processor such as UpdateAttribute which is in a stopped/disabled state. This will hold the data in the dataflow until manually intervention can be taken to identify the issue and either reroute the data back in to the flow once corrected or discard the data. If there is concern over available space in your NiFi Content repository, i would some processor to write it out to a different error file location using putFile, PutHDFS, PutSFTP, etc... Hope this helps, Matt ... View more

@Raj B Process groups can be nested inside process groups and with the granular access controls NiFi provides i may not be desirable for every user who has access to the NiFi Ui to be able to access all processors or the specific data those processors are using. So in addition to your valid example above, you may want to create stove pipe dataflows based off different input ports where only specific users are allowed view and modify to the stove pipe dataflow they are responsible for. While you of course can have flowfiles from multiple upstream sources feed into a single input port and then use a routing type processor to split them back out to different dataflows, it can be easier just to have multiple input ports to achieve the same affect with less configuration. Matt ... View more

@Avish Saha The behavior you should be seeing here is that the mergeContent processor will take the first incoming FlowFile it sees and add it to bin 1. It will then continue to attempt to add additional FlowFiles to Bin 1 until either 1000 total FlowFiles have been added or the min size has reached 1 MB. Now lets say bin 1 has grown to 1000KB (just shy of 1 MB) and the next FlowFile would cause that bin to exceed the max group size of 1 MB. In this case that File would not be allowed to go into bin 1 and would be the first file to start bin 2. Now bin 1 hangs around because the min requirement of 1 MB has not been met and neither max entries or max group size has been met either. So you can see it is possible to fill all 5 of your bins without meeting your very tightly configured thresholds. So what happens when a next FlowFile will not fit in any of the 5 existing bins? The mergeContent processor will merge to oldest bin to free room to start a new bin. So what I am assuming here is you are seeing few or no files that are exactly 1 MB. Thanks, Matt ... View more