question Re: Heavy CPU usage by NiFi Java process in Archives of Support Questions (Read Only)

Heavy CPU usage by NiFi Java process

Raj_B — Fri, 16 Sep 2022 10:16:29 GMT

Hi All,

I have a simple NiFi Dataflow on my local Windows machine, but NiFi's Java process utilization on my machine is spiking high; I was not sure what's causing the CPU to spike, so I closed the NiFi instance (aborted the running run-nifi.bat batch file and killed the Java Process in Task Manager) and restarted NiFi, and found no change, CPU still spiking; this spiking is without Starting any of the NiFi Processors; also, uncommented these 3 properties in bootstrap.conf, but still no help;

java.arg.7=-XX:ReservedCodeCacheSize=256m

java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m

java.arg.9=-XX:+UseCodeCacheFlushing

I then started to check system diagnostics, which showed that both "FlowFile repository storage usage" and "Content repository storage usage" are at 86% (271 GB of 315 GB); I don't know what those numbers mean, but I'm puzzled; I don't even have that much free space (271 GB) on my machine, what do these numbers represent; and how do I flush these and get a clean start, thinking that doing so would help the CPU spiking issue.

Any help is appreciated.

Thanks,

Raj

Re: Heavy CPU usage by NiFi Java process

pvillard — Sun, 01 May 2016 16:59:56 GMT

Hi Raj,

Have you flows running with NiFi? If processors were running when stopping NiFi, they will be running when NiFi starts.

If yes, could you tell us what processors are running? Also, could you have a look into the logs (nifi-app.log) to copy/paste here anything that could be relevant?

Regarding repositorie: the Content Repository holds the content for all the FlowFiles in the system, and the FlowFile repository keeps track of the attributes and current state of each FlowFile in the system. (I'd recommend having a look here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html)

Re: Heavy CPU usage by NiFi Java process

TimothySpann — Mon, 02 May 2016 03:17:09 GMT

is there anything in the logs?

Re: Heavy CPU usage by NiFi Java process

Raj_B — Mon, 02 May 2016 07:46:00 GMT

When I experienced this issue, all NiFi Processors are stopped; the only thing was there was one connection that was queued up; trying to empty the queue was not working (a progress bar would show up, but it would just stay at 0% and not progress)

Checked the nifi-app.log file; "java.lang.OutOfMemoryError: GC overhead limit exceeded" is in the log quite a lot.

Since I'm running NiFi on my local laptop, I unzipped/installed NiFi again in any another folder and exported/imported the dataflow as a template from the old folder to the new folder; that seems to have resolved the issue I was facing.

Re: Heavy CPU usage by NiFi Java process

TimothySpann — Mon, 02 May 2016 08:51:28 GMT

that's strange, what JDK is on my machine

Re: Heavy CPU usage by NiFi Java process

pvillard — Mon, 02 May 2016 14:40:35 GMT

How many flow files in the queue?

It sounds like you used a processor generating large amounts of FlowFiles and your computer was not able to handle so much. You may want to consider back pressure features, or changing the scheduling of processors.

Re: Heavy CPU usage by NiFi Java process

james_jones — Tue, 03 May 2016 00:47:08 GMT

I had something like this happen on my 16G macbook pro when I switched from 0.4.1 to 0.6.0 and reused the repository directories. It was just an experimental instance, so I abandoned the original repos and started from scratch, except for importing a few flows. I have not had the problem since. The odd thing in my case is that it was fine in 0.4.1 but not in 0.6.0.

Re: Heavy CPU usage by NiFi Java process

Raj_B — Tue, 03 May 2016 22:39:37 GMT

I am reading flat files from a Unix directory, there are about 8 million files, the files are small, 1Kb-2Kb; but, that may have been the problem; thanks for the suggestion, I will look into back pressure and scheduling; thanks Pierre

Re: Heavy CPU usage by NiFi Java process

Raj_B — Tue, 03 May 2016 22:41:24 GMT

It's JDK 1.8