Reply
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

directory write-up sequence hdfs

I have seen hadoop write to frst 3  filesystems first and fill 100% ,while ignoring other where more space is available.

/dev/sdb1 916G 870G 17M 100% /data/01
/dev/sdc1 916G 870G 17M 100% /data/02
/dev/sdd1 916G 870G 37M 100% /data/03
/dev/sde1 916G 654G 217G 76% /data/04
/dev/sdf1 916G 656G 215G 76% /data/05
/dev/sdg1 916G 666G 204G 77% /data/06
/dev/sdh1 916G 662G 208G 77% /data/07
/dev/sdi1 916G 667G 203G 77% /data/08
/dev/sdj1 916G 657G 214G 76% /data/09
/dev/sdk1 916G 657G 213G 76% /data/10
/dev/sdl1 916G 655G 215G 76% /data/11
/dev/sdm1 916G 660G 211G 76% /data/12
/dev/sdn1 916G 642G 228G 74% /data/13
/dev/sdo1 916G 649G 222G 75% /data/14
/dev/sdp1 916G 656G 214G 76% /data/15
/dev/sdq1 916G 659G 212G 76% /data/16
/dev/sdr1 916G 654G 216G 76% /data/17
/dev/sds1 916G 660G 211G 76% /data/18
/dev/sdt1 916G 659G 211G 76% /data/19
/dev/sdu1 916G 652G 218G 76% /data/20
/dev/sdv1 916G 657G 214G 76% /data/21
/dev/sdw1 916G 655G 215G 76% /data/22

 

i have same situation on all DNs. for NMs i have created container dir on all the FS just to make sure NM process jobs.

but why hdfs do not write to other FS with more space available. is it becuase of round robin policy ?

if i use volume policy will it help and solve this problem .

 

Please guide .

 

Posts: 519
Topics: 14
Kudos: 90
Solutions: 45
Registered: ‎09-02-2016

Re: directory write-up sequence hdfs

@MSharma

 

Pls add the required file system (if it is not already added) under 

 

CM -> HDFS -> Configuration -> DataNode Data Directory

 

and restart the service. Also you may need to rebalance after that

Highlighted
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: directory write-up sequence hdfs

they are already there DN data dir .
but still see first 3 dir getting fill to 100%
Announcements