Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

High CPU usage when running ExecuteSQL

avatar
Explorer

I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)
Bootstrap.conf was set like the following:
java.arg.2=-Xms2g
java.arg.3=-Xmx24g
java.arg.13=-XX:+UseG1GC <not sure if needed even>

ExecuteSQL is set like the following:

LiranJP_0-1712584357473.png

LiranJP_1-1712584491417.png

 


The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.
Seems like the ExecuteSQL is consuming high CPU & RAM. 
The java.arg.3=-Xmx24g is the upper limit for the RAM?
How can I control the CPU?



1 REPLY 1

avatar
Master Collaborator

Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL.

Loading a large table in one go with ExecuteSQL could result in high heap usage.

on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required. 

Thank you