How to replicate a file in NiFi

Why do you want to do that ? NiFi cluster is not like HDFS where data will be replicated across nodes NiFi is not a storage system. 

What NiFi provides as , let's say you have 3 node cluster and you receive 100 files on one node assuming Processor which one is receiving file running on Primary node,now you want to processor these all 100 files in distributed manner using all 3 nodes , then you need to use Load Balancing at queue connection to distribute 100 file on 3 nodes , 33/33/34 files on each node then rest further downstream processing will be done on all 3 nodes.  Example flow design flow be, ListSFTP (Primary)-->Connection set to LB at queue tion --> FetchSFTP, in this use case ListSFTP will list the file details on Primary node then FetchSFTP will fetch the file from SFTP server on All 3 nodes in a distributed manner.

Thanks 

Hi,

There is a processor called DuplicateFlowFile where you configure it with the Number of Copies that you wish to have. I assume in your case if you want to process it on multiple nodes then the number of copies should be n-1 (where n is number of nodes and -1 because you still have the original flow file). In the downstream queue for this processor success relationship make sure to configure the queue Load Balance Strategy to "Round Robin" where each flowfile will be send to a different node. Hope that helps.

I'm not seeing the load balancing being set on the success relationship of the DuplicateFlowFile. That will basically mean that the PutFile processor will be executed on each node (as many as the flowfile coming out of the duplicate processor.

Current setup is as below.  

Basically to reiterate the concept.  We wish to receive the single file uploaded on a single data node (1 of 7), process that through NiFi to present an individual copy of that file on every data node.  Currently I am getting a multiple created by the duplicateflowfile processor but this is not being placed on the individual nodes.

Ok, so you got the copy file in NiFi using List/FetchFile then you changed the new directory location where it has to be written using UpdateAttribute then you have used DuplicateFlowFile to create duplicates, till now what is happening since ListFile is running on Primary node (And it should be running on Primary ) rest all downstream flow will be processed/computed on Primary that why you see DuplicateFlowFile creating duplicated on node which is Primary in NiFi Cluster. I do not see any issue here. If you wish to have flow files on All NiFi nodes redistributed after DuplicateFlowFile then you need to set Load Balancing at queue connection on the connection "Round Robin " between DuplicateFlowFile & PutFile, by doing this duplicates flow files will be distributed among nifi nods in round-robin manner one by one and will be written by PutFile back on each nifi nodes where files got distributed after  DuplicateFlowFile.  Now one question you mention 7 HDFS data nodes, does that means you have  NiFi service running as well on the same 7 data nodes ? and having 7 node nifi cluster? in that case, only all 7 nodes will receive each file copy. I see 

@Tryfan 

You mention this file comes in daily.  You also mention that this file arrives through a load-balancer so you don't know which node will receive it.  This means you can't configure your source processor for "primary node" only execution as you have done in your shared sample flow with the ListFile.   As Primary Node only, the elected primary node will be the only node that executes that processor.  So if the source file lands on any other node, it would not get listed.

You could handle this flow in the following manor:

GetFile ---> (7 success relationships) PostHTTP or InvokeHTTP (7 of these with one configured for each node in your cluster cluster)
ListenHTTP --> UpdateAttribute --> PutFile

So in this flow, no matter which node receives your source file, the GetFile will consume it. It will then get cloned 6 times (7 copies then exist) with one copy of the FlowFile getting routed to 7 unique PostHttp processors.  Each of these targets the ListenHTTP processor listening on each node in your cluster. That ListenHTTP processor will receive all 7 copies (one copy per node) of the original source file.  Then use the UpdateAttribute to set your username and location info before the putFile which place each copy in the desired location on the source node.  

If you add or remove nodes from your cluster, you would need to modify this flow accordingly which is a major downside to such a design.  Thus the best solution is still one where the source file is placed somewhere all nodes can retrieve it from so it scales automatically.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt