Thank you for your answer, The second hash is performed after a PutHDFS and not a ListHDFS (I have edited my post, sorry for the mistake). If I understand you well, this check is not useful because PutHDFS and FetchFile processors are already able to catch corruption errors reliably? I would compare both hashes with NiFi expression language (:equals function) inside a RouteOnAttribute processor. ... View more

@umang s Based on your flow design above, it looks like you are trying to route FlowFiles by comparing attribute between two different FlowFiles? That will not work. NiFi is looking for both ${temp_array} and ${category} to exist on same flowfile being evaluated by the RouteOnAttribute processor. ... View more

@Derek Calderon - Sorry to hear that. I did share this HCC link with a few devs I know if they have time to assist. - Thanks, Matt ... View more

@Derek Calderon - Short answer is no. The ExecuteSQL processor is written to write the output to the FlowFile's content. - There is an alternative solution. You have some processor currently feeding FlowFiles to your ExecuteSQL processor via a connection. My suggestion would be to feed that same connection to two different paths. The first connection feeds to a "MergeContent" processor via a funnel and the second feeds to your "ExecuteSQL" processor. The ExecuteSQL processor performs the query and retrieves the data you are looking for writing it to the content of the FlowFile. You then use a processor like "ExtractText" to extract that FlowFIles new content to FlowFile Attributes. Finally you use a processor like "ModifyBytes" to remove all content of this FlowFile. Finally you feed this processor to the same funnel as the other path. The MergeContent processor could then merge these two flowfiles using the "Correlation Attribute Name" property (assuming "filename" is unique, that could be used), min/max entries set to 2, and "Attribute Strategy" set to "Keep All Unique Attributes". The result should be what you are looking for. - Flow would look something like following: Having multiple identical connections does not trigger NiFi to write the 200 mb of content twice to the the content repository. a new FlowFile is created but it points to the sam content claim. New content is only generated when the executeSQL is run against one of the FlowFiles. So this flow does not produce any additional write load on the content repo other then when the executeSQL writes its output which i am assuming is relatively small? - Thank you, Matt ... View more

Ok, thanks Matt, this template helps me ... View more

perfect, thanks guys! ... View more

@Sami Ahmad Users who are not authorized for specific NiFi components (process groups, processors, input/output ports, etc...) will only see "ghost" processors. Those ghost processors will not contain any information on the face of them the reveals their type or configuration. - - Unless authorized, user also will not be able to list queues, perform provenance queries, etc on FlowFiles that traverse these components. - There are good reasons why NiFi does not hide these ghost processor completely: 1. All components operate within the same resource constraints of a single JVM. It is important that all user can identify areas of any flow that may be backlogging or have large queues, as these things may impact their own dataflows. While the can not see the actual data or processor details, they can observe FlowFile stat info as you can see in above screenshot. 2. If User1 built a dataflows only user 1 could see, User2 may end up building a dataflow directly on top of it. While functionally NiFi does not care about overlapping components, user3 who logs in and has access to both user1 and user2 flows would. It would make it very hard for user 3 to follow the canvas in such a case. - Thanks, Matt ... View more

@Vladislav Shcherbakov - The ListFTP processor generates the following FlowFile attributes on each 0 byte FlowFile it generates: so you would typically use NiFi's Expression Language (EL) to define values for those properties dynamically per each processed FlowFile: For example: The property "Remote File" on the FetchFTP processor set a value of "${filename}". With each FlowFile received it will return the value assigned to this attribute and use it to retrieve the correct FlowFiles content from the target FTP server. - Thank you, Matt ... View more

@rajat puchnanda Here is screenshot showing the menus and icons that go along with the above explanation: Thanks, Matt ... View more