I am trying to figure out the difference between df -h in linux and hdfs dfs -df -h / output. dfs output showing 44Tb across all cluster but root partition is not having the data it is always showing very less usage on each data node.
hdfs@hdfs-xxxxx-xxxxxxx:~$ hdfs dfs -df -h / Filesystem Size Used Available Use% hdfs://hdfs-xxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:8020 44.8 T 20.6 T 6.8 T 46% hdfs@hdfs-xxxxxx-xxxxxx:~$ where as df -h hdfs@hdfs-XXXXX-XXXX:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 32G 12K 32G 1% /dev tmpfs 6.3G 320K 6.3G 1% /run /dev/sda1 2.0T 8.8G 1.9T 1% / XXXXXXXXXXXXXXXXXXXXXXX 2.0T 1.8T 142G 93% /hadoop hdfs@hdfs-XXXXX-XXXXX:~$ dfsadmin report ================== Decommission Status : Normal Configured Capacity: 3785823326208 (3.44 TB) DFS Used: 1893513980859 (1.72 TB) Non DFS Used: 1370250273861 (1.25 TB) DFS Remaining: 302107447808 (281.36 GB) DFS Used%: 50.02% DFS Remaining%: 7.98% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B)
May I know where is the utilization is going on and why it is not showing on / partition.
Here hdfs dfs -df -h refers to the HDFS filesystem and displays the amount of disk space available on the file system containing.
df -h refer to the local filesystem with respect to that particular server or node.
df displays the amount of disk space available on the file system containing each file name argument. If no file name is given, the space available on all currently mounted file systems is shown
Always hdfs dfs -df -h and hdfs dfsadmin -report should give you the same information as below:
[hdfs@abcd ~]$ hdfs dfs -df -h Filesystem Size Used Available Use% hdfs://hdplab 139.5 G 9.1 G 119.2 G 7% [hdfs@abcd ~]$ hdfs dfsadmin -report Configured Capacity: 149775056896 (139.49 GB) Present Capacity: 137747146752 (128.29 GB) DFS Remaining: 127948359680 (119.16 GB) DFS Used: 9798787072 (9.13 GB) DFS Used%: 7.11% Under replicated blocks: 48 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0
Hope this helps you.