Support Questions
Find answers, ask questions, and share your expertise

NiFi:ListenHTTP - errors when multiple client connections

Solved Go to solution

NiFi:ListenHTTP - errors when multiple client connections

Explorer

Hello,

I tested ListenHTTP on a single node under some load generated by jmeter.

 

I observed that for the one or two client threads the ListenHTTP works fine, however,
having 10 client threads leads to 60% request refused by 503,Service Unavailable.

 

Is there any parameter, which helps to come accross the issue?
Also the component does not indicate any problems in the GUI.

 

Best regards
Jaro

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi:ListenHTTP - errors when multiple client connections

@Jarinek    Some infofmation about your nifi configuration will help be more accurate.  For example: min/max ram, number of cores, disk configuration, etc..  Information about your flow is important too. The processor/queue back pressure, concurrency, and run time all effect the performance you need. 

 

Without this information, it sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config).   You should look to increase concurrency, increase queue size and back pressure based on # of flowfiles moving through. your data flow.  You should also inspect the min/max thread counts as these have a major impact on performance.   All of these items will be seriously limited with a single node, so be mindful of your expectations.

 

If you can I would recommend a small 3 node nifi cluster to evaluate nifi performance in a better test environment where you can really turn up the performance and distribute the work load across 3 nodes.   With 3 times as many cores & ram you can make better use of min/max thread count, increase concurrency much higher, and you should see the stability you are expecting.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

View solution in original post

2 REPLIES 2
Highlighted

Re: NiFi:ListenHTTP - errors when multiple client connections

@Jarinek    Some infofmation about your nifi configuration will help be more accurate.  For example: min/max ram, number of cores, disk configuration, etc..  Information about your flow is important too. The processor/queue back pressure, concurrency, and run time all effect the performance you need. 

 

Without this information, it sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config).   You should look to increase concurrency, increase queue size and back pressure based on # of flowfiles moving through. your data flow.  You should also inspect the min/max thread counts as these have a major impact on performance.   All of these items will be seriously limited with a single node, so be mindful of your expectations.

 

If you can I would recommend a small 3 node nifi cluster to evaluate nifi performance in a better test environment where you can really turn up the performance and distribute the work load across 3 nodes.   With 3 times as many cores & ram you can make better use of min/max thread count, increase concurrency much higher, and you should see the stability you are expecting.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

View solution in original post

Highlighted

Re: NiFi:ListenHTTP - errors when multiple client connections

Explorer

It sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config)

 

Correct assessment. It has showed that the pipeline was not properly sized for the amount of data, which lead to a back-pressure in the ingest component