Support Questions
Find answers, ask questions, and share your expertise

Nifi: sizing parameters - Need Recommendation

New Contributor

We have a requirement - where 1000 calls per second for our HandleHTTPRequest processor. Could you please suggest the sizing parameters for the requirement. We have a three node cluster currently.

1. How may cores would be needed? [per node]

2. Suggested memory per node + suggested heap max. [per node]

3. Container size and concurrent tasks setting recommendation.- for HandleHTTPRequest & HandleHTTPResponse processors.

4. Any other sizing parameter to be implemented?

 

+ We get too many open files error when we try bump up the load. What would be the best nuumber to be set for /proc/sys/fs/file-max.

 

Please suggest.

 

 

2 REPLIES 2

Super Collaborator

Sizing is a somewhat complex topic. The documentation offers basic guidelines, but for specific flows it is always hard. 

 

If you already have a working cluster, my most practical recommendation would be to just try (perhaps start small and build up) to see what load it can currently handle. It is quite possible that your current setup can facilitate your new usecase as well.

 

If you find that you need to expand your footprint, I would recommend you to contact your Cloudera contact person for more detailed guidance.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl

New Contributor
Thank you for the document.