@TRSS_Cloudera
Your use case is not completely clear to me.
Each Node in a NiFi cluster executes its own copy of the dataflow against its own set fo FlowFiles (FlowFiles are what the NiFI components execute upon). NiFi components can be processors, controller services, reporting tasks, input/output ports, RPG, etc. Each node maintains its own set of repositories. Two of those repositories (flowfile_repository and content_repository) hold the parts that make up a FlowFile.
In a NiFi cluster a node will always get elected as the Cluster Coordinator or Primary Node (sometimes one node is elected for both these roles) Which node is elected to either role can change at anytime.
Your GenerateFlowFIle processor you have configured to execute on "Primary Node" only will produce FlowFile(s) only on the currently elected primary node. From your description, you dod not cover how your dataflow writes the files to the server that you will then run an ExecuteStreamCommand Python script on.
- What is the best way to handle producing files that can be accessed by all nodes?
Answer: Since each node operates on its own FlowFiles, one node will not have access to FlowFiles on the other nodes. A clearer use case as to why you would want every node processing the same FlowFile might be helpful here. - Is there a way to specify the node for a process will be run on? (using “run on primary” is not working as the primary node cycles over the process)
Answer: Only processors that are responsible for creating the FlowFile should ever be scheduled to execute on the "Primary Node". Any processor that accepts and inbound connection should always be executing on all nodes. So if Node A is current Primary Node and a FlowFile is produced by a primary node only configured processor, the FlowFile would still be processed downstream in the dataflow even if a primary node change happens.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
