How to calculate cluster size as well as no of node requirement

Hi @Ashik ,

A good rule of thumb for the amount of HDFS storage required is 4 x the raw data volume. HDFS triple replicates data and then we need some headroom in the system which is why it is 4 x rather than 3 x . This formula is just a rough guide and can change for example if you compress the data on HDFS. You need to factor other data processing that you might do into this calculation. For example, if you built data marts on top of the raw data - that is additional data volume and then you have organic data growth over a period of time.

Regarding cluster topology, there are some guidelines here:

https://docs.cloudera.com/documentation/enterprise/5/latest/topics/cm_ig_host_allocations.html

Regarding best practice for cluster sizing, those are here:

https://docs.cloudera.com/documentation/other/reference-architecture/topics/ra_bare_metal_deployment.html#concept_lzl_vkl_f2b

Regarding hardware recommendations, those are given here:

https://docs.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html

I would recommend that you have:

• 3 x Master Nodes (for high availability)
• N x Data Nodes (where N is number based on the storage capacity of the data nodes). You need a minimum of N=3 for triple replication of data and I would recommend N >= 5 for a production system. The more data nodes that you have and the more disks there are in each of those data nodes the higher the performance of your system will be because of the distributed throughput and higher disk I/O.
• 1 x Utility Node / Management Node
• 1 x Gateway Node

Regards,

Steve