@jfs912
You should not be configuring your NiFi with larger then necessary heap. Doing so just leads to very long stop-the-world garbage collection events. The simple flow you have described would use very little heap memory.
- So you have a 5 node cluster and each nodes has files in some local directory that each node is pulling from?
- Is that local directory a mounted directory that is mounted to all nodes or each nodes has its own set of files in the local directly from which getFile is pulling from?
- Are you seeing backpressure being applied on any of the connections between your processors? When backpressure is being applied to the upstream processor, NiFi will not schedule that upstream processor until that backpressure is removed.
- If you can tolerate some latency in your dataflow, you can get better throughput performance with some processors by increasing the Run Duration as well.
- Dataflow design best practices and designs can also improve performance and better load distribution across all the nodes in your cluster. You want to minimize as much as possible one node doing bulk of the work load.
Adjusting concurrent tasks has multiple elements to it.
- What is the current CPU load average on each of yoru 5 servers? First need to know if there is capacity to run more parallel threads.
- How large is the configured timer driven thread pool in NiFi? It is from this configured thread pool that all concurrent tasks used by processor components comes from. If this pool is small, adding more concurrent tasks to processors will improve nothing. Ability to increase the size of this thread pool is dependent on node's cpu load average. Thread pool is also applied per node. So when set to 10 that is 10 threads per each node in your 5 node cluster.
- If cpu load average is not high and you increase the size of the Timer Driven Thread pool, you'll want to make small incremental changes to the concurrent tasks on processor and monitor impact on CPU load average.
Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
@jfs912
You should not be configuring your NiFi with larger then necessary heap. Doing so just leads to very long stop-the-world garbage collection events. The simple flow you have described would use very little heap memory.
- So you have a 5 node cluster and each nodes has files in some local directory that each node is pulling from?
- Is that local directory a mounted directory that is mounted to all nodes or each nodes has its own set of files in the local directly from which getFile is pulling from?
- Are you seeing backpressure being applied on any of the connections between your processors? When backpressure is being applied to the upstream processor, NiFi will not schedule that upstream processor until that backpressure is removed.
- If you can tolerate some latency in your dataflow, you can get better throughput performance with some processors by increasing the Run Duration as well.
- Dataflow design best practices and designs can also improve performance and better load distribution across all the nodes in your cluster. You want to minimize as much as possible one node doing bulk of the work load.
Adjusting concurrent tasks has multiple elements to it.
- What is the current CPU load average on each of yoru 5 servers? First need to know if there is capacity to run more parallel threads.
- How large is the configured timer driven thread pool in NiFi? It is from this configured thread pool that all concurrent tasks used by processor components comes from. If this pool is small, adding more concurrent tasks to processors will improve nothing. Ability to increase the size of this thread pool is dependent on node's cpu load average. Thread pool is also applied per node. So when set to 10 that is 10 threads per each node in your 5 node cluster.
- If cpu load average is not high and you increase the size of the Timer Driven Thread pool, you'll want to make small incremental changes to the concurrent tasks on processor and monitor impact on CPU load average.
Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
