I have a flow where we are receiving HTTP post request on Nifi cluster. I am using ListenHttp processor for the same. Once I receive the json payload I am splitting array of json into individual json objects (flowfile).
These individual jsons are then processed by series of downstream processors. (ETL)
But when we get a sudden spike of requests I see connector heading towards splitJson is growing in queue size. I can also see that CPU is in 20s for all nodes.
Attaching some screenshots of queue configuration and splitJson processor configuration.
How can I optimise the processing of this processor.
How many nodes are in your NiFi?
How many cores per node?
I see you have concurrency set to 90 and in the screen shot ~2400 active threads.
At this point, your tuning is going to be at the Nifi cluster level (ram, cores, more nodes). You will also need to evaluate Garbage Collection on the NiFI nodes.
Another thing to check out would be these admin settings:
Congrats, that is a substantial nifi cluster. Before other collaborators get into comment, we still want some more details. @MattWho can probably help better than I can. Let's make sure he has more than enough context.
Few more questions: