Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

why hdfs dfs -df -h / output and df -h output are different?


I am trying to figure out the difference between df -h in linux and hdfs dfs -df -h / output. dfs output showing 44Tb across all cluster but root partition is not having the data it is always showing very less usage on each data node.

hdfs@hdfs-xxxxx-xxxxxxx:~$ hdfs dfs -df -h /
Filesystem                                                    Size    Used  Available  Use%
hdfs://hdfs-xxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:8020  44.8 T  20.6 T      6.8 T   46%

where as df -h 

hdfs@hdfs-XXXXX-XXXX:~$ df -h
Filesystem              Size  Used Avail Use% Mounted on
udev                     32G   12K   32G   1% /dev
tmpfs                   6.3G  320K  6.3G   1% /run
/dev/sda1               2.0T  8.8G  1.9T   1% /
XXXXXXXXXXXXXXXXXXXXXXX  2.0T  1.8T  142G  93% /hadoop

dfsadmin report 

Decommission Status : Normal
Configured Capacity: 3785823326208 (3.44 TB)
DFS Used: 1893513980859 (1.72 TB)
Non DFS Used: 1370250273861 (1.25 TB)
DFS Remaining: 302107447808 (281.36 GB)
DFS Used%: 50.02%
DFS Remaining%: 7.98%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)

May I know where is the utilization is going on and why it is not showing on / partition.



@sysadmin CreditVidya

Here hdfs dfs -df -h refers to the HDFS filesystem and displays the amount of disk space available on the file system containing.

df -h refer to the local filesystem with respect to that particular server or node.

df displays the amount of disk space available on the file system containing each file name argument. If no file name is given, the space available on all currently mounted file systems is shown

Always hdfs dfs -df -h and hdfs dfsadmin -report should give you the same information as below:

[hdfs@abcd ~]$ hdfs dfs -df -h
Filesystem            Size   Used  Available  Use%
hdfs://hdplab  139.5 G  9.1 G    119.2 G    7%

[hdfs@abcd ~]$ hdfs dfsadmin -report
Configured Capacity: 149775056896 (139.49 GB)
Present Capacity: 137747146752 (128.29 GB)
DFS Remaining: 127948359680 (119.16 GB)
DFS Used: 9798787072 (9.13 GB)
DFS Used%: 7.11%
Under replicated blocks: 48
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0 

Hope this helps you.