Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How many requests ListenHTTP processor can handle at a time?

Highlighted

How many requests ListenHTTP processor can handle at a time?

New Contributor

Hello community,


My NIFI process flow gets its input from ListenHTTP processor. Potentially thousands of POST requests could be made at the same time. I wonder how many requests ListenHTTP processor can handle at a time.

My next question would be how can I scale up the data injection mechanism (specifically for receiving more POST requests at a time).

Thank you,
Arash

5 REPLIES 5

Re: How many requests ListenHTTP processor can handle at a time?

ListenHTTP creates an embedded Jetty server with a thread pool of 200 threads, so presumably a single ListenHttp could handle whatever you could do with a single Jetty instance with 200 threads.

To scale out you would have a NiFi cluster with a ListenHTTP on each node and a load balancer in front and you'd have your clients POST to the load balancer URL.

Re: How many requests ListenHTTP processor can handle at a time?

New Contributor

Actually there is no limit on the Jetty's number of threads and I was able to hit Jetty (NIFI's ListenHTTP processor) with 500 simultaneous requests (tested with JMeter).

Re: How many requests ListenHTTP processor can handle at a time?

New Contributor

Hi Bryan,

Thanks for your answer. I wonder how increasing concurrency would play out. If I set ListenHTTP processor's concurrency to 5, would I have 5 instances of jetty and 1000 threads in total?

Re: How many requests ListenHTTP processor can handle at a time?

The Jetty server is running separately and creating flow files from the POST requests and then placing the flow files in an internal list/queue, and then when the processor executes it takes a flow file out of the internal queue and transfers it to the success relationship. So the number of concurrent tasks is the number of threads processing the internal queue, but there is only ever one Jetty server per processor.

Re: How many requests ListenHTTP processor can handle at a time?

New Contributor

Thank you Bryan.