@iriszhuhao  This might be a good use case for using the FlowFile Concurrency  and Outbound Policy configuration option on a process group.  FlowFile concurrency allows you to place a portion of your dataflow into a process group and be able to control how the initial FlowFile or batch of FlowFiles is allowed to enter that process group for processing.  The Outbound Policy controls when the FlowFiles being processed in that process group will be released to processor(s) downstream of that process group.  Downstream components of the process group will not receive FlowFiles from the process group until all FlowFiles within the process group have either been auto-terminated or queued up to one or more output ports.  When the outbound policy is met, the FlowFile(s) are released downstream and the Process group's FlowFile concurrency then allows for next batch processing.  So it might makes sense to place the portion of your dataflow comprised of your nine concurrent branches in this bounded process group and downstream you have your your ExecuteSQLRecord processor call your final procedure now that you know all branches have completed. Above solves your problem with not all nine branches always being used.  If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@hegdemahendra  Curious about the differences between your prod and uat environments here. 1. Same number of nodes in each environments NiFi cluster? 2. Same exact configurations on the ConsumeKafka processor (except consumer group ID) ? 3. Screenshot of scheduling tab for consumeKafka processor? 4. Versions of NiFi (Apache, CFM, HDF) used in both environments? 5. Any observed rebalancing or consumer group related exceptions in the logs? Having more consumers in a consumer group than the number of partitions can lead to constant rebalance occurring.   The number of consumers in the consumer group is calculated by multiplying the number of nodes in your NiFi cluster by the number of concurrent tasks configured on the ConsumeKafka processor.  So if you have a 3 node cluster, you should have only 1 concurrent task, so number of consumers is equal to or less the the number of partitions.   Thanks, Matt ... View more

@PriyankaMondal  I am not clear by you statement: if Nifi processor (any processor within a process group) stops suddenly due to load/any other issue You are saying you see a NiFi processor transition to a stopped state unexpectedly? This should never happen. Or are you saying the processor seems to dtop processing FlowFiles even though it is currently in a running/started state? NiFi queues FlowFiles in connection between processor components.  A FlowFile is not removed from the inbound connection to a processor component until that FlowFile has been successfully processed by the consuming processor.  The FlowFile consist of two parts: 1. FlowFile attributes/metadata that is persisted in the NiFi flowfile_repository. 2. FlowFile content persisted within claims inside the content_repository.  To protect from data loss these repositories should be using protected storage such as RAID. So if NiFi were to suddenly crash or server itself crash, when NiFi is restarted on that down node it will load its flow and then load the FlowFile back in to the connections.  Processing will begin again against those FlowFiles by downstream processor component. NiFi's design favors data duplication over data loss ir order to avoid data loss posibilities. For example:  Let's assume that a NiFi processor completed execution against a FlowFile resulting in writing something out to an external endpoint.  in response to that successful operation the processor would then move the FlowFile from the inbound connection to some a downstream relationship.  If NiFi were to crash in that very moment before the FlowFile was moved, on startup the same FlowFile would load in the inbound connection and get processed again. Also keep in mind that you are running 3 node NiFi cluster and within a NiFi cluster each connected node runs its own copy of the flow, its own set of repositories, and  its own local state.  So each node is unaware of the FlowFiles being processed by another node in the same cluster. Generally speaking when you have a processor that shows active threads indicator on it and zeroed out stats, you either have a very long running thread or a hung thread (only examination of serious of thread dumps can make the determination.  Most commonly this is a resource utilization problem.  But could also be dataflow design issue, client library issue, or network issue. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@ALWOSABY  The putHDFS processor has properties for changing the Remote Owner and Remote Group, but in order to use these properties certain condition must be met: If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more