Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

High CPU usage when running ExecuteSQL

avatar
Explorer

I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)
Bootstrap.conf was set like the following:
java.arg.2=-Xms2g
java.arg.3=-Xmx24g
java.arg.13=-XX:+UseG1GC <not sure if needed even>

ExecuteSQL is set like the following:

LiranJP_0-1712584357473.png

LiranJP_1-1712584491417.png

 


The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.
Seems like the ExecuteSQL is consuming high CPU & RAM. 
The java.arg.3=-Xmx24g is the upper limit for the RAM?
How can I control the CPU?



1 REPLY 1

avatar
Master Collaborator

Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL.

Loading a large table in one go with ExecuteSQL could result in high heap usage.

on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required. 

Thank you