Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi splitJson is becoming bottleneck

Highlighted

Nifi splitJson is becoming bottleneck

New Contributor

Hello,

 

I have a flow where we are receiving HTTP post request on Nifi cluster. I am using ListenHttp processor for the same. Once I receive the json payload I am splitting array of json into individual json objects (flowfile).

These individual jsons are then processed by series of downstream processors. (ETL)

But when we get a sudden spike of requests I see connector heading towards splitJson is growing in queue size. I can also see that CPU is in 20s for all nodes.

Attaching some screenshots of queue configuration and splitJson processor configuration.

 

How can I optimise the processing of this processor.

Screen Shot 2020-01-03 at 11.29.16 PM.pngSplitJson propertiesScreen Shot 2020-01-03 at 11.29.06 PM.pngConnection details1Screen Shot 2020-01-03 at 11.28.54 PM.pngConnection details2Screen Shot 2020-01-03 at 11.26.57 PM.pngSettingsScreen Shot 2020-01-03 at 11.25.52 PM.pngFlow

3 REPLIES 3

Re: Nifi splitJson is becoming bottleneck

Super Collaborator

How many nodes are in your NiFi?

 

How many cores per node?

 

 

I see you have concurrency set to 90 and in the screen shot ~2400 active threads.

 

At this point,  your tuning is going to be at the Nifi cluster level (ram, cores, more nodes).  You will also need to evaluate Garbage Collection on the NiFI nodes.  

 

Another thing to check out would be these admin settings:  

 

https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-max-thread-pools-and-process...

 

 

Re: Nifi splitJson is becoming bottleneck

New Contributor

This nifi cluster has 30 nodes with 16 cores and 32GB memory.

Re: Nifi splitJson is becoming bottleneck

Super Collaborator

Congrats, that is a substantial nifi cluster.   Before other collaborators get into comment, we still want some more details.  @MattWho can probably help better than I can.  Let's make sure he has more than enough context.

 

Few more questions:

 

  1. Can you describe your disk arrangement?
  2. Can you share your Thread Count settings (reference link in my previous reply)?
  3. What are the nifi min/max memory settings?
  4. Can you comment on any additional steps you have taken from known NiFI Performance Tuning?

 

https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-p...

 

Don't have an account?
Coming from Hortonworks? Activate your account here