@Benjamin Bouret - It is common for the cluster Coordinator and Primary Node to change from time to time in a NiFi cluster, so you need to careful when designing your flows that utilize processors running "Primary node" only to make sure processing can still continue when a switch occurs. - I am going to assume since you got duplicates here that the local directory you have your GetFile processor pointing at is mounted across all your NiFi nodes. In order to avoid duplicates you will need to use processors that support state. The GetFile processor is one of our original processors that was developed before state management was put in place. It has been deprecated in favor of the newer listFile and FetchFile processors. The ListFile processor has the ability to store state either local to each node (not shared for cases where each node is pulling from its own non shared directory) or cluster state (state is stored in zookeeper where same processor on every node has access to it). Cluster state here would allow you to run this processor against a shared mount to all you nodes in "Primary node" only setup. If primary node changes the new primary node will start this processor and pull the last known recorded cluster state before performing a new listing. This should greatly reduce the likelihood of seeing duplicates. - NiFi will favor duplicate data over lost data. So there will still exist a small window of opportunity where duplication could occur. For example original primary node ingested data but some network issue for example prevented last state to be written to zookeeper. The new node would then not get the most current state which may result in duplication. - The list/fetch processor model also allows you to spread the workload across your cluster more easily. A flow would consist of: listFile (Scheduled primary node only) --> Remote Process group (configured to point back at cluster to redistributed listed files) ---> fetchFile ( running on all nodes to retrieve content of listed files) --> rest of flow... - Thanks, Matt - If you found this answer addressed your question, please take a moment to login to the forum and click "accept" on the answer. ... View more

@Naeem Ullah Khan I am not sure that i am completely clear on what you are trying to do... - "run this process as a cron job on a linux machine independent of NiFi" - Are you asking how to run these processors outside of a NiFi installation? If so, that is not possible. The NiFi core (Controller) handles the scheduling and management of the resources (CPU, heap, repositories, etc...) NiFi flows can not be exported to independent running processes. - You can install NiFi on your linux server(s) and even configure the run schedule as a cron. You can also create a template of yoru flow on one NiFi, download it, and then import it in to another NiFi installation. - Thanks, Matt - If you found this answer addressed your question, please take a moment to login to the forum and click "accept" on the answer. ... View more