Nifi splitJson is becoming bottleneck

[#1]

Hello,

I have a flow where we are receiving HTTP post request on Nifi cluster. I am using ListenHttp processor for the same. Once I receive the json payload I am splitting array of json into individual json objects (flowfile).

These individual jsons are then processed by series of downstream processors. (ETL)

But when we get a sudden spike of requests I see connector heading towards splitJson is growing in queue size. I can also see that CPU is in 20s for all nodes.

Attaching some screenshots of queue configuration and splitJson processor configuration.

How can I optimise the processing of this processor.

SplitJson propertiesConnection details1Connection details2SettingsFlow

[#2]

How many nodes are in your NiFi?

How many cores per node?

I see you have concurrency set to 90 and in the screen shot ~2400 active threads.

At this point,  your tuning is going to be at the Nifi cluster level (ram, cores, more nodes).  You will also need to evaluate Garbage Collection on the NiFI nodes.  

Another thing to check out would be these admin settings:  

https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-max-thread-pools-and-process...

[#3]

This nifi cluster has 30 nodes with 16 cores and 32GB memory.

[#4]

Congrats, that is a substantial nifi cluster.   Before other collaborators get into comment, we still want some more details.  @MattWho can probably help better than I can.  Let's make sure he has more than enough context.

Few more questions:

1. Can you describe your disk arrangement?
2. Can you share your Thread Count settings (reference link in my previous reply)?
3. What are the nifi min/max memory settings?
4. Can you comment on any additional steps you have taken from known NiFI Performance Tuning?

https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-p...