question Re: HDFS is almost full 90% but data node disks are around 50% in Archives of Support Questions (Read Only)

HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Sun, 18 Aug 2019 09:05:42 GMT

hi all

we have ambari cluster version 2.6.1 & HDP version 2.6.4

from the dashboard we can see that HDFS DISK Usage is almost 90%

but all data-node disk are around 90%

so why HDFS show 90% , while datanode disk are only 50%

/dev/sdc                   20G   11G  8.7G  56% /data/sdc
/dev/sde                   20G   11G  8.7G  56% /data/sde
/dev/sdd                   20G   11G  9.0G  55% /data/sdd
/dev/sdb                   20G  8.9G   11G  46% /data/sdb

is it problem of fine-tune ? or else

we also performed re-balance from the ambari GUI but this isn't help

Re: HDFS is almost full 90% but data node disks are around 50%

jsensharma — Sun, 18 Aug 2019 09:05:34 GMT

@Michael Bronson

The HDFS dashboard metrics widget "HDFS Disk Usage" shows: \The percentage of distributed file system (DFS) used, which is a combination of DFS and non-DFS used.

So can you just put your mouse over the "HDFS Disk Usage" widget and then see what is the different values do you see there for "DFS Used" , "non DFS Used" and Remaining. You should see something like following:

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Sun, 18 Aug 2019 09:05:26 GMT

@Jay , this is what we got: ( so this is like 88% used ) , and regarding my question , how it can be 88% when disk are ~50%

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 12:25:59 GMT

@Jay , I want to note that we have 4 datenode machines ( 4 workers machines ) and each worker have 4 disks with 20G

Re: HDFS is almost full 90% but data node disks are around 50%

jsensharma — Tue, 04 Sep 2018 12:26:42 GMT

@Michael Bronson

What do you see when you run the following command?

# su - hdfs -c " hdfs dfsadmin -report | grep 'DFS Used%'

(OR)

Please check the "DFS Used" shown in the NameNode UI to verify if ambari is shoiwng the same data or different? : http://$ACTIVE_NAMENODE:50070/dfshealth.html#tab-overview

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 12:29:39 GMT

@jay , we got the following results


 su - hdfs -c " hdfs dfsadmin -report | grep 'DFS Used%' "
DFS Used%: 87.38%
DFS Used%: 88.48%
DFS Used%: 87.00%
DFS Used%: 84.70%
DFS Used%: 87.93%

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 12:43:09 GMT

@Jay another info from my datanode about the disk use



datanode1
47% disk1
64% disk2
48% disk3
49% disk4


datanode2
44%
53%
61%
44%


datanode3
55%
46%
45%
91%


datanode4
63%
45%
49%
46%

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 13:16:21 GMT

@Jay do you need other info?

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Sun, 18 Aug 2019 09:05:18 GMT

@Jay , according to http://$ACTIVE_NAMENODE:50070/dfshealth.html#tab-overview

we get

Re: HDFS is almost full 90% but data node disks are around 50%

jsensharma — Tue, 04 Sep 2018 17:05:40 GMT

@Michael Bronson

As "HDFS Disk Usage" shows: The percentage of distributed file system (DFS) used, which is a combination of DFS and non-DFS used.

The NameNode commands/UI shows that the DFS Used is around 87.06% and Non DFS Used is 0%

So which is almost same which ambari is showing like almost 88% (DFS + Non DFS Usage) so there seems to be no contradiction to me.

Please let us know what is the value you are expecting.

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 17:16:09 GMT

@jay , yes I agree , but how it can be the datanode disk are with capacity of 50% and HDFS show 88% ? , and why we not used all the size of datanode disks ? , I am really not understant that

Re: HDFS is almost full 90% but data node disks are around 50%

rajasuba_s — Tue, 04 Sep 2018 17:16:11 GMT

HDFS originally splits and stores the data in blocks. Each block is 64MB or 128MB by default based on your HDFS version. Consider a file which is of size (2MB) is stored in a block. The remaining 62MB (considering the default size to be 64MB) is not used by HDFS. Which means here the HDFS used space is 64MB but the actual hard disk used space is 2MB.

Hope this is relevant to what you are asking.

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 17:32:18 GMT

@Rabbit , so in our case based on that 88% is HDFS uses , the only one option to be with more HDFS space is to add adisk in each datanode ? am I right ?

Re: HDFS is almost full 90% but data node disks are around 50%

Shelton — Tue, 04 Sep 2018 17:36:05 GMT

@Michael Bronson

If you have heavily deleted files then it could still be in the .Trash.

HDFS trash is just like the Recycle Bin. Its purpose is to prevent you from unintentionally deleting something. You can enable this feature by setting this property:

fs.trash.interval

with a number greater than 0 in core-site.xml. After the trash feature is enabled, when you remove something from HDFS by using the rm command, files or directories will not be wiped out immediately; instead, they will be moved to a trash directory (/user/${username}/.Trash, see example).

$ hdfs dfs -rm -r /user/bob/5gb file
15/09/18 20:34:48 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
Moved: ‘hdfs://hdp2.6/user/bob/5gb’ to trash at: hdfs://hdp2.6/user/bob/.Trash/Current

If you want to empty the trashor just delete the entire trash directory use the HDFS command line utility to do that:

$ hdfs dfs -expunge

Or use the -skipTrash

hdfs dfs -rm -skipTrash /path/to/file/you/want/to/remove/permanently

Can you check that the hidden directory .Trash ?

Re: HDFS is almost full 90% but data node disks are around 50%

rajasuba_s — Tue, 04 Sep 2018 17:54:59 GMT

Yes possibly @Michael Bronson. You could also check "Trash" files size as @Geoffrey Shelton Okot suggests.

In general it is advised not to store too many small files in HDFS. In general, HDFS is good for storing large files.

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 18:06:20 GMT

@Geoffrey , yes we already did

hdfs dfs -expunge

and still HDFS is around 88%

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Tue, 04 Sep 2018 19:17:25 GMT

@Geoffrey So the last thing that we can to do is to add disk on the datanode , or add data-node machine , what your opinion ?

Re: HDFS is almost full 90% but data node disks are around 50%

hdpcda — Tue, 04 Sep 2018 22:13:45 GMT

Hi,

HDFS is full one of reason is Replication Factor, You’re comparing the “before replication".

you have 4 data nodes. How much space does each of the 4 data nodes have ?

Re: HDFS is almost full 90% but data node disks are around 50%

mike_bronson7 — Wed, 05 Sep 2018 00:26:10 GMT

@Ganesh

we have 4 datanode machines and each datanode have 4 disks , and each disk have 20G , let me know if this info that you want ?

Re: HDFS is almost full 90% but data node disks are around 50%

hdpcda — Wed, 05 Sep 2018 02:35:21 GMT

Total 320 GB , what is the replication setting 2 or 3 and check initially how many disks you used for hdfs( how much hdfs space created)