@Raj B If you have FlowFiles arriving via multiple input ports and then passing through some common set of components downstream from them, there is no way to tell by looking at the FlowFile in a given queue which input port it originated from. Input and output ports within a process group do not create provenance events either since they do not modify the FlowFiles in anyway. The only way an input port or output port would generate a Provenance event is if it was on the root canvas level since inout would generate a "create" event and output ports would create a "drop" event. Provenance will show a lineage for a FlowFile which will show any processor which routed or modified the FlowFile in some way. So by looking at the details of the various events in the provenance lineage graph you can see where the FlowFile traversed through your Flow. However, as I stated not all processors create provenance events. When you query provenance, you can access the lineage for any of the query results by clicking the show lineage icon: A lineage graph for the specific FlowFile will then be created and displayed: The red dot show the event the lineage was calculate from. Every circle is another event in this particular FlowFiles life. You can right click on any of the events to view the details of the event including which specific processor in your flow produced that event. Thanks, Matt ... View more

@Raj B Not all Errors are equal. I would avoid lumping all failure relationships into the same error handling strategy. Some Errors are no surprise, can be expected to occur on occasion, and may be a one time thing that resolves itself. Lets use your example above.... The putHDFS processor is likely to experience some failure over time do to events outside of NiFi's control. For example, let say a file in the middle of transferring to HDFs when the network connection is lost. NIFi would in turn route that FlowFile to failure. If that failure relationship had been routed back on the putHDFS, it would have likely been successful on the subsequent attempt. A better error handling strategy in this case may be to build a simple error handling flow that can be used when the type of failure might lead to self resolution. So here you see Failed FlowFiles enter at "data", they are then checked for a failure counter and if one does not exist it is created and set to 1. If it exists, it is incremented by 1. The check recount count will continue to pass the file to "retry" until the same file has been seen x number of times. "Retry" would be routed back to the source processor of the failure. after x attempts the counter is reset, an email is sent, and the file is place in some local error directory for manual intervention. https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1&modificationDate=1433271239000&api=v2 The other scenario is where the type of failure is not likely to ever correct itself. Your mergeContent processor is a good example here. If the processor failed to merge some FlowFiles, it is extremely likely to happen again, so there is little benefit in looping this failure relationship back on the processor like we did above. In this case you may want to route this processors failure to a putEmail processor to notify the end user of the failure and where it occurred in the dataflow. The success of the putEmail processor may just feed another processor such as UpdateAttribute which is in a stopped/disabled state. This will hold the data in the dataflow until manually intervention can be taken to identify the issue and either reroute the data back in to the flow once corrected or discard the data. If there is concern over available space in your NiFi Content repository, i would some processor to write it out to a different error file location using putFile, PutHDFS, PutSFTP, etc... Hope this helps, Matt ... View more

@Raj B Process groups can be nested inside process groups and with the granular access controls NiFi provides i may not be desirable for every user who has access to the NiFi Ui to be able to access all processors or the specific data those processors are using. So in addition to your valid example above, you may want to create stove pipe dataflows based off different input ports where only specific users are allowed view and modify to the stove pipe dataflow they are responsible for. While you of course can have flowfiles from multiple upstream sources feed into a single input port and then use a routing type processor to split them back out to different dataflows, it can be easier just to have multiple input ports to achieve the same affect with less configuration. Matt ... View more