Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DataNode Volume Choosing Policy

DataNode Volume Choosing Policy

Explorer

Hi All,

 

we found that DataNode Volume Choosing Policy parameter not work as our expectation.  our datanode had varrying storage size, due to data growth now the datanode that had storage size smaller than others always got bad health status. it seem like still using round robin configuration so the files that thrown to HDFS will be spread to all datanode. below are our configurations related with datanode 

 

dfs.datanode.fsdataset.volume.choosing.policy=Round Robin

dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 10GB

dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75

 

 

is there any other configuration that related the HDFS block placing that i missed so DataNode Volume Choosing Policy not working properly ?

 

 

regards,

i9um0

 

3 REPLIES 3

Re: DataNode Volume Choosing Policy

Explorer

Hi All,

 

just wrong type to dfs.datanode.fsdataset.volume.choosing.policy=Round Robin the correct one is available space. :D

 

 

Regards,

i9um0

Highlighted

Re: DataNode Volume Choosing Policy

Same situation, what are the parameters to set for datanodes having different disk sizes.

For example,

one datanode has - disk space 7 TB

other datanode has - disk space 15TB

 

Any help is highly appreciated.

Re: DataNode Volume Choosing Policy

Super Collaborator

You should use 

Available Space instead of Round Robin for
dfs.datanode.fsdataset.volume.choosing.policy
 
Don't have an account?
Coming from Hortonworks? Activate your account here