Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Minimum hardware requirements for HDP

Highlighted

Minimum hardware requirements for HDP

Contributor

Minimum Hardware requirements for HDP for Production ?

3 REPLIES 3

Re: Minimum hardware requirements for HDP

Guru

@mel mendoza

Its not that simple to answer your question directly. There are lot of variables which comes to play when you are choosing the hardware for hadoop.

Few things are:

Volume of historical data which will be pushed to HDFS

Volume of data which will be processed on a daily basis

Needs of HDFS/ how are you planning to use HDFS.

Frequency of data

Type of data Hot/warm/cold

Data scaling

understanding the hardware w.r.t needs of data usage.

To Initial start with its decent to start with 10 nodes with 512 GB of RAM. Moving forward based on the variables you can always scale up. Hadoop is basically designed for such data scaling. So rather then implementing the Hadoop in production at one shot, its better to have it as different migration. For example start to push the cold/warm data first ( probably small amount or data which will not impact other process) then start pushing the data and jobs gradually into HDFS thereby its easier for you to understand the needs of a component and if needed you can scale up easily with Hadoop.

Check this link and this

Re: Minimum hardware requirements for HDP

Contributor

Hi,

It is recommended to have at-least 10 nodes minimum for Production level cluster, also provide the details like initial data ingestion, tools that are going to be used in cluster! So that it would be easy to recommend the RAM, Disk, No of nodes and others !!!

Please refer below article !

https://community.hortonworks.com/articles/16763/cheat-sheet-and-tips-for-a-custom-install-of-horto....

Minimum Hardware Requirements for HDP 2.5:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-installation/content/meet-m...

Cheers,

Ram

Re: Minimum hardware requirements for HDP

Guru

@mel mendoza

Its not that simple to answer your question directly. There are lot of variables which comes to play when you are choosing the hardware for hadoop.

Few things are:

Volume of historical data which will be pushed to HDFS

Volume of data which will be processed on a daily basis

Needs of HDFS/ how are you planning to use HDFS.

Frequency of data

Type of data Hot/warm/cold

Data scaling

understanding the hardware w.r.t needs of data usage.

To Initial start with its decent to start with 10 nodes with 512 GB of RAM. Moving forward based on the variables you can always scale up. Hadoop is basically designed for such data scaling. So rather then implementing the Hadoop in production at one shot, its better to have it as different migration. For example start to push the cold/warm data first ( probably small amount or data which will not impact other process) then start pushing the data and jobs gradually into HDFS thereby its easier for you to understand the needs of a component and if needed you can scale up easily with Hadoop.

Please check out these links:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_cluster-planning-guide/content/ch_hardwa...

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_cluster-planning-guide/content/hardware-...