We have a requirement - where 1000 calls per second for our HandleHTTPRequest processor. Could you please suggest the sizing parameters for the requirement. We have a three node cluster currently.
1. How may cores would be needed? [per node]
2. Suggested memory per node + suggested heap max. [per node]
3. Container size and concurrent tasks setting recommendation.- for HandleHTTPRequest & HandleHTTPResponse processors.
4. Any other sizing parameter to be implemented?
+ We get too many open files error when we try bump up the load. What would be the best nuumber to be set for /proc/sys/fs/file-max.
Sizing is a somewhat complex topic. The documentation offers basic guidelines, but for specific flows it is always hard.
If you already have a working cluster, my most practical recommendation would be to just try (perhaps start small and build up) to see what load it can currently handle. It is quite possible that your current setup can facilitate your new usecase as well.
If you find that you need to expand your footprint, I would recommend you to contact your Cloudera contact person for more detailed guidance.