- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU usage when running ExecuteSQL
- Labels:
-
Apache NiFi
Created ‎04-08-2024 06:55 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have Nifi 1.25 on Ubuntu EC2 (m6i.2xlarge) = (8CPU/32GB RAM)
Bootstrap.conf was set like the following:
java.arg.2=-Xms2g
java.arg.3=-Xmx24g
java.arg.13=-XX:+UseG1GC <not sure if needed even>
ExecuteSQL is set like the following:
The flow is getting about 50-60 tables from mySQL and start creating flow files from them , upload to S3 and then copy into Redshift.
Seems like the ExecuteSQL is consuming high CPU & RAM.
The java.arg.3=-Xmx24g is the upper limit for the RAM?
How can I control the CPU?
Created ‎04-15-2024 06:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL.
Loading a large table in one go with ExecuteSQL could result in high heap usage.
on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required.
Thank you
