hdfs data disk size is exceeding 90% threshold while rest of the disks(in the same server) are about 55%

[#1]

Hi all,

Can you please guide me how to troubleshoot why one of the disks in one data node is exceeding its size than the others.

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00 1.8T 3.1G 1.7T 1% / tmpfs 127G 264K 127G 1% /dev/shm /dev/sda1 485M 38M 422M 9% /boot /dev/sdb1 1.8T 956G 785G 55% /data01 /dev/sdc1 1.8T 964G 777G 56% /data02 /dev/sdd1 1.8T 960G 781G 56% /data03 /dev/sde1 1.8T 931G 810G 54% /data04 /dev/sdf1 1.8T 962G 779G 56% /data05 /dev/sdg1 1.8T 944G 796G 55% /data06 /dev/sdh1 1.8T 945G 796G 55% /data07 /dev/sdi1 1.8T 1.6T 192G 90% /data08 /dev/sdj1 1.8T 934G 806G 54% /data09 /dev/sdk1 1.8T 930G 811G 54% /data10 /dev/sdl1 1.8T 940G 800G 55% /data11 /dev/mapper/VolGroup00-LogVol04 5.0G 696M 4.0G 15% /home /dev/mapper/VolGroup00-LogVol03 9.9G 470M 8.9G 5% /opt /dev/mapper/VolGroup00-LogVol05 5.0G 150M 4.6G 4% /tmp /dev/mapper/VolGroup00-LogVol02 20G 1.9G 17G 11% /var

[#2]

Hi aengineer!

Thanks so much for the information. Is the fix good for any HDP release. I want to use it on HDP 2.1 & 2.4.2

Also, if i need to rebalance the disks, do i really need to decommission it or just stop the Datanode and start after a while...??

Thanks in advance..

[#3]

As far as I know, we have not backported this change to HDP 2.1 or 2.4.2. There is nothing technically preventing us from doing so; Disk balancer does not depend on any of the newer 3.0 features.

> , if i need to rebalance the disks, do i really need to decommission it or just stop the Datanode and start after a while...??

Stopping datanodes does not change the disk usage, so if one disk is over utilized, some writes will fail, that is when the datanode picks that disk(it generally uses a round-robin allocation scheme). So you need to make sure data is similarly distributed in each of the disks. That is what diskBalancer does for you, it computes how much to move based on each disk type ( Disk, SSD etc). When you decommission a node all the blocks of this machine is moved to other nodes. Then you can go into this node and make sure all the data disks are empty and add the node back. Then you will need to run balancer -- not disk balancer -- to move data back to this node. Sorry it is so painful.

[#4]

Hi, thanks so much for the information. But for now I deleted the hdfs /tmp folder contents which were lying there for a long time. This freed up about 500GB of space on hdfs in total, and that particular disk went down to 82% from 90%. How is it possible? The other disk also which had the same issue went down to 82%. My question is: did the disk usage go down just because of deleting the /tmp folder or does the disk size fluctuate because of some other running jobs too??

Also, I thought mapreduce utilizes local disk for storing intermediate data, so what is actually stored in hdfs tmp directory? I assumed this is where the intermediate data is stored, which is utilizing hdfs space.

Thanks again in advance.