HDFS is almost full 90% but data node disks are around 50%

[#1]

hi all

we have ambari cluster version 2.6.1 & HDP version 2.6.4

from the dashboard we can see that HDFS DISK Usage is almost 90%

but all data-node disk are around 90%

so why HDFS show 90% , while datanode disk are only 50%

/dev/sdc                   20G   11G  8.7G  56% /data/sdc
/dev/sde                   20G   11G  8.7G  56% /data/sde
/dev/sdd                   20G   11G  9.0G  55% /data/sdd
/dev/sdb                   20G  8.9G   11G  46% /data/sdb

is it problem of fine-tune ? or else

we also performed re-balance from the ambari GUI but this isn't help

[#2]

Yes possibly @Michael Bronson. You could also check "Trash" files size as @Geoffrey Shelton Okot suggests.

In general it is advised not to store too many small files in HDFS. In general, HDFS is good for storing large files.

[#3]

@rabbit s

-1. I totally disagree.

HDFS calculates exactly what it used. Please don't confuse it.

[#4]

@Karthik Palanisamy : I'm sorry to be wrong. Consider two blocks (64 MB, 64MB) and only I've used only (2MB, 3MB) in each block. Means can I reuse the remaining (123MB) to another one block of 64MB?

[#5]

Yes, you can use remaining space to other blocks.

[#6]

so what is the final conclution , why we have a gap between what the disks size and the HDFS as displayed on the amabri dasborad ?

[#7]

As I said earlier, It's hard to tell you the exact cause without reviewing namenode and datanode log regarding disk registration. As we see in the UI that configured capacity is 154 GB which means it registered only two disks from each datanode.

If you don't have any concern then share your logs which should be after the service restart.

I still waiting for your reply to my previous question,

Did you validated in local machine, hdfs-site.xml without ambari?

# grep dfs.datanode.data.dir -A1 /etc/hadoop/conf/hdfs-site.xml

[#8]

this is the file: , and its look fine

<name>dfs.datanode.data.dir</name>
      <value>/data/sdb/hadoop/hdfs/data,/data/sdc/hadoop/hdfs/data,/data/sdd/hadoop/hdfs/data,/data/sde/hadoop/hdfs/data</value>
--
      <name>dfs.datanode.data.dir.perm</name>
      <value>750</value>

[#9]

let me know if you have some conclusions , as you saw the configuration in HDFS and in the XML is correct , so I show you the real status and disk are corectly configured in HDFS ,,

[#10]

@Michael Bronson

If you have heavily deleted files then it could still be in the .Trash.

HDFS trash is just like the Recycle Bin. Its purpose is to prevent you from unintentionally deleting something. You can enable this feature by setting this property:

fs.trash.interval 

with a number greater than 0 in core-site.xml. After the trash feature is enabled, when you remove something from HDFS by using the rm command, files or directories will not be wiped out immediately; instead, they will be moved to a trash directory (/user/${username}/.Trash, see example).

$ hdfs dfs -rm -r /user/bob/5gb file
15/09/18 20:34:48 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
Moved: ‘hdfs://hdp2.6/user/bob/5gb’ to trash at: hdfs://hdp2.6/user/bob/.Trash/Current 

If you want to empty the trashor just delete the entire trash directory use the HDFS command line utility to do that:

$ hdfs dfs -expunge 

Or use the -skipTrash

hdfs dfs -rm -skipTrash /path/to/file/you/want/to/remove/permanently 

Can you check that the hidden directory .Trash ?

[#11]

@Geoffrey , yes we already did

hdfs dfs -expunge

and still HDFS is around 88%