High CPU usage when running ExecuteSQL

LiranJP — Mon, 08 Apr 2024 13:55:23 GMT

I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)
Bootstrap.conf was set like the following:
java.arg.2=-Xms2g
java.arg.3=-Xmx24g
java.arg.13=-XX:+UseG1GC <not sure if needed even>

ExecuteSQL is set like the following:

The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.
Seems like the ExecuteSQL is consuming high CPU & RAM.
The java.arg.3=-Xmx24g is the upper limit for the RAM?
How can I control the CPU?

Re: High CPU usage when running ExecuteSQL

ckumar — Mon, 15 Apr 2024 13:20:23 GMT

Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL.

Loading a large table in one go with ExecuteSQL could result in high heap usage.

on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required.

Thank you

question High CPU usage when running ExecuteSQL in Support Questions

High CPU usage when running ExecuteSQL

Re: High CPU usage when running ExecuteSQL