Nifi process consuming >100%CPU while executing dataflow

[#1]

i'm new to NiFi and have developed simple dataflow which would read 3 files from on-prem and upload into S3 bucket. i am using ListFile, FetchFile and PutS3 processor groups for this case.

When i trigger dataflow, CPU consumption is spiking >100% and NiFi gets crashed.After few minutes. it automatically comes down to normal. This is happening frequently. My onprem server is Linux and has 8 CPUs. Any idea what's going on  with this.

[#2 · Accepted solution]

A couple things you should do:

1) set the schedule of listfile to 1 min or 5 min.  If it’s 0 sec it’s always running which isn’t necessary during testing.

2) tail /var/log/nifi/nifi-app.log while trouble shooting the flow to see errors. Address them individually.

I also suspect the Nifi node does not have enough resources (ram/cores/disk) so look into 3 dedicated nifi nodes with enough core and ram and disk configure for each repository type (see documentation) to allow you to operate Nifi in a stable manner.

[#3 · Accepted solution]

@Gubbi please share you nifi info:

How many nodes?

How much ram and how many cores per node?

What is min/max memory setting in NiFi?

Have you done anything with configuring nifi for performance?  For example

1. Increasing Min/Max Ram?
2. Disk Partitioning?
3. Changing the Max Thread Count in Admin->Controller Services?
4. Setting Concurrent Tasks in processor scheduling tab?

[#4]

@stevenmatison  :

1. Changing the Max Thread Count in Admin->Controller Services? Max Timer Driven--16 , Max Event Driven --1 
2. Setting Concurrent Tasks in processor scheduling tab? - Set to 1 as default
3. Increasing Min/Max Ram? - Set to default

Below is my Linux configuration :

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
Stepping: 4
CPU MHz: 2294.609
BogoMIPS: 4589.21
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-7

[#5]

@Gubbi   I think your ListFile proc is still executing 0 sec.   Reference our private message.