Thanks @mclark for your input. As suggested we implemented the data flow as shown below: 1)The standalone flow(This is where the single source file arrive).The RPG in this flow refers to the cluster i.e NCM url 2)The NCM of the cluster has flow as shown below: 3)In this approach were facing the error as shown below as getting only a subset of the records in HDFS 4)So to avoid the situation we used a MergeContent processor to merge the flowfiles since we were splitting them before loading them to HDFS 5)We configured the MergeContent in the way as shown below: But even after this implementation we are not getting all the records in HDFS. The source file has 10000000 records and approximately 5000000 records should go to each HDFS directory. But we are getting around 1000000 records in each Target and the error as shown below in the PutHDFS processors. We are getting the same error as mentioned in the snapshot attached with the point 3 above. Are we missing something very intrinsic here? Is there something wrong with the design? We are using a 3 node cluster with NCM and 2 slave nodes. And the source file is coming to a standalone server. Let me know if you need any other information. Any inputs would be appreciated. Regards, Indranil Roy ... View more

@mclark We have a single large(in TB's) flowfile coming to a standalone node.We want to distribute the processing.Is it a good approach to split the file into multiple smaller files using SplitText processor so that the processing is distributed to the remaining clusters. In such a case we are considering the the flow given below: In the NCM of the cluster input->RouteText->PutHDFS In the standalone processor that has the incoming flow file ListFile->FetchFile->SplitText->UpdateAttribute->RPG(NCM url) Does this set up ensure the processing to be distributed? ... View more

Hi @Pierre Villard We are using NIFI 0.6.0 To answer your second question a subset of the whole file is coming to the ReplaceText processor based on the condition in the RouteText processor.Say we have 100 records and some 50 records satisfies the condition so 50 records will come to the processor. The regular expression we are using is (.+)\|(.+)\|(.+).... Where (.+) is repeated n of times based on the number of columns in the flowfile. So as per your observation we should be using ^(.+)\|(.+)\|(.+)....$ Any other suggestion to improve the performance? ... View more

@mclark 1)We are talking about a single file in TB. 2)There is a single file and the processing should be distributed. 3)The file are in the local directory. So is it a good idea? ... View more

Hi @mclark Thanks for the alternate approach that you suggested.It could be helpful in my case. Say in the scenario mentioned above we have a single input file of size in the order of TB's.If we use a ListSFTP/FetchSFTP processor in the way you mentioned to distribute the fetching of data: Do we need to establish a SFTP channel between every slave node of the cluster and the remote server that houses the source file for this approach to work? Is it a good idea to use SFTP to fetch the file considering the size of the file will be in TB's? What are the parameters on which the the performance of the fetch using ListSFTP/FetchSFTP will depend? ... View more

Thanks @Bryan Bende for your input As per your suggestion I have included two different flows as given below: Flow1 ===== In the standalone instance ListFile->FetchFile->Outputport In the NCM of the cluster RPG(Standalone NIFI Instance)-->Routetext->PutHDFS(This is the processing done in the main cluster) Flow2 ===== In the standalone instance ListFile->FetchFile->Input port of RPG(NIFI Cluster URL of the NCM) In the NCM of the cluster Input port->RouteText->OutputPort Which according to you is the correct flow.I can understand that the fetching of the source file cannot be distributed if the file is not shared but which of the flows will be apt to distribute the part of the processing done inside the cluster? Regards, Indranil Roy ... View more

Thanks a lot.It works. ... View more

It was in stopped state.As I already pointed out I was able to update other properties.This is specific to "Search Value" property in replaceText processor. ... View more