Hadoop Hardware guide

[#1]

Hi , Since the introduce of Hadoop 3.0 and beyond, we have EC , intra disk balancer, and Ozone and traditionally we have storage policy for HDFS

just wondering, given that the hardware world have changed a lot , eg SSD/Intel Optane Cache and 10G/25G/100G NIC etc.

do we have an up-to-date hardware sizing and physical layout for best practices for those up-to-date hardware with our technology?

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_cluster-planning/content/server-node.1.h...

this content is just out dated so much

[#2]

Bumb!

I have the exact same question. Can't find any hardware recommendations in the HDP 3 docs.

,

Bump!

I'm wondering the same thing.

[#3]

Such as this https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/cluster-planning/content/hardware-recommend...

[#4]

Hi Yan:

Yes this documentation is terribly outdated.  However when we are building our clusters for our Cloudera customers now the hardware configuration is not so much different than previous versions.  There are more cores and larger disk drives but we are not doing anyhting exotic like Optane at this point.  I do think that with tiering it is probably time to consider an SSD and spinning disk platform.  Maybe consider NVMe drives but not really sure you are going to need that performance. I would bump up memory (at least 256GB per node at this point) before moving to NVMe. As for networking dual 10GigE is standard.  I think we will probably shoot through 25G / 40G and move right to 100G if pricing conintues to come down. 

Alex