Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS + different disk size nodes + data volume choosing policy

HDFS + different disk size nodes + data volume choosing policy

Lets say we have the following cluster with the following details

 

10 node HDFS cluster, and 4 are of disk size - 10 TB and 6 node of disk size - 1TB

 

On Hadoop - 2.6, cloudera - 5.8 , we have ability to change the default from round robin to available space in case the disks on data node machine are with different size

 

Example ( in case of cloudera cluster )


look at dfs.datanode.fsdataset.volume.choosing.policy. By default this is set to round-robin but since you have an asymmetric disk setup you should change it to available space.

 

Capture.PNG

 

Since we have hortonwoks HDP cluster version 2.6.5 ,

 

we are searching the same ability

So we search in AMBARI HDFS CONFIG


But we not found the configuration about round-robin / available space in AMBARI HDFS CONFIG

 

Dose HDP 2.6.5 ambari cluster can give this ability ?

 

The goal is to balance the data on all disks consider that some disks are small then others

 

other post with the same discussion - https://community.cloudera.com/t5/Support-Questions/General-guidelines-and-best-practices-for-tuning...

Capture1.PNG

Michael-Bronson
Don't have an account?
Coming from Hortonworks? Activate your account here