Skip to main content
LiranJP
LiranJP
Silver ⭐️⭐️
April 8, 2024
Question

High CPU usage when running ExecuteSQL

I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)
Bootstrap.conf was set like the following:
java.arg.2=-Xms2g
java.arg.3=-Xmx24g
java.arg.13=-XX:+UseG1GC <not sure if needed even>

ExecuteSQL is set like the following:

LiranJP_0-1712584357473.png

LiranJP_1-1712584491417.png

 


The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.
Seems like the ExecuteSQL is consuming high CPU & RAM. 
The java.arg.3=-Xmx24g is the upper limit for the RAM?
How can I control the CPU?



1 reply

ckumar
Platinum ⭐️
ckumar
Platinum ⭐️
April 15, 2024

Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL.

Loading a large table in one go with ExecuteSQL could result in high heap usage.

on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required. 

Thank you

Translate
    Terms & ConditionsAccessibility statement