Support Questions

Find answers, ask questions, and share your expertise

NiFi:ListenHTTP - errors when multiple client connections

Contributor

Hello,

I tested ListenHTTP on a single node under some load generated by jmeter.

 

I observed that for the one or two client threads the ListenHTTP works fine, however,
having 10 client threads leads to 60% request refused by 503,Service Unavailable.

 

Is there any parameter, which helps to come accross the issue?
Also the component does not indicate any problems in the GUI.

 

Best regards
Jaro

1 ACCEPTED SOLUTION

@Jarinek    Some infofmation about your nifi configuration will help be more accurate.  For example: min/max ram, number of cores, disk configuration, etc..  Information about your flow is important too. The processor/queue back pressure, concurrency, and run time all effect the performance you need. 

 

Without this information, it sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config).   You should look to increase concurrency, increase queue size and back pressure based on # of flowfiles moving through. your data flow.  You should also inspect the min/max thread counts as these have a major impact on performance.   All of these items will be seriously limited with a single node, so be mindful of your expectations.

 

If you can I would recommend a small 3 node nifi cluster to evaluate nifi performance in a better test environment where you can really turn up the performance and distribute the work load across 3 nodes.   With 3 times as many cores & ram you can make better use of min/max thread count, increase concurrency much higher, and you should see the stability you are expecting.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

View solution in original post

2 REPLIES 2

@Jarinek    Some infofmation about your nifi configuration will help be more accurate.  For example: min/max ram, number of cores, disk configuration, etc..  Information about your flow is important too. The processor/queue back pressure, concurrency, and run time all effect the performance you need. 

 

Without this information, it sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config).   You should look to increase concurrency, increase queue size and back pressure based on # of flowfiles moving through. your data flow.  You should also inspect the min/max thread counts as these have a major impact on performance.   All of these items will be seriously limited with a single node, so be mindful of your expectations.

 

If you can I would recommend a small 3 node nifi cluster to evaluate nifi performance in a better test environment where you can really turn up the performance and distribute the work load across 3 nodes.   With 3 times as many cores & ram you can make better use of min/max thread count, increase concurrency much higher, and you should see the stability you are expecting.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

Contributor

It sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config)

 

Correct assessment. It has showed that the pipeline was not properly sized for the amount of data, which lead to a back-pressure in the ingest component 

 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.