Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Limit Concurrent Tasks on Nifi Cluster

Highlighted

Limit Concurrent Tasks on Nifi Cluster

New Contributor

Has some way to set how many concurrent tasks is needed for a processor?

Example: with a cluster with 10 machines, I want to set 6 concurrent tasks along the cluster

It's possible?

1 REPLY 1

Re: Limit Concurrent Tasks on Nifi Cluster

Master Guru
@Allan Batista

-

Each NiFi node in your 10 node cluster is running with its own copy of the flow.xml and operating against its own set of FlowFiles. Concurrent tasks setting on a component (for example processor) is applied to every node. In order for a NiFi node to execute a component it must have the ability to request a thread from the controller, so it must have at least 1 concurrent task. So setting 1 concurrent task (default) on a processor means that every node can run that processor with one concurrent task (so 10 potential threads in operation across cluster). Threads are not shared between nodes.

-

Also keep in mind that setting 6 concurrent tasks on a single component (60 equivalent across c10 node cluster) does not mean that that component on each node will always use all 6. If the volume of data being processed by the component does not warrant using more then 1 or 2 concurrent tasks, the other concurrent tasks will not be utilized.

-

This article may help here:
https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor....

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.