Reply
Explorer
Posts: 14
Registered: ‎09-03-2014

DataNode Volume Choosing Policy

Hi All,

 

we found that DataNode Volume Choosing Policy parameter not work as our expectation.  our datanode had varrying storage size, due to data growth now the datanode that had storage size smaller than others always got bad health status. it seem like still using round robin configuration so the files that thrown to HDFS will be spread to all datanode. below are our configurations related with datanode 

 

dfs.datanode.fsdataset.volume.choosing.policy=Round Robin

dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 10GB

dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75

 

 

is there any other configuration that related the HDFS block placing that i missed so DataNode Volume Choosing Policy not working properly ?

 

 

regards,

i9um0

 

Explorer
Posts: 14
Registered: ‎09-03-2014

Re: DataNode Volume Choosing Policy

Hi All,

 

just wrong type to dfs.datanode.fsdataset.volume.choosing.policy=Round Robin the correct one is available space. :D

 

 

Regards,

i9um0

Explorer
Posts: 10
Registered: ‎06-23-2017

Re: DataNode Volume Choosing Policy

Same situation, what are the parameters to set for datanodes having different disk sizes.

For example,

one datanode has - disk space 7 TB

other datanode has - disk space 15TB

 

Any help is highly appreciated.

Expert Contributor
Posts: 322
Registered: ‎01-25-2017

Re: DataNode Volume Choosing Policy

You should use 

Available Space instead of Round Robin for
dfs.datanode.fsdataset.volume.choosing.policy
 
Announcements