https://community.cloudera.com/t5/Archives-of-Support-Questions/Non-DFS-Used-is-reported-much-bigger-than-bash-df-h-reports/m-p/62624#M72458 <P>This problem is HDFS-9530 which has a fix in CDH-5.9.0.</P><P>&nbsp;</P><P>Bouncing the DN instances cleared this issue manually until we upgrade.</P><P>&nbsp;</P> Fri, 08 Dec 2017 22:19:12 GMT ScottChris 2017-12-08T22:19:12Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Non-DFS-Used-is-reported-much-bigger-than-bash-df-h-reports/m-p/62569#M72457 <P>Seeing this issue on all data nodes.</P><P>&nbsp;</P><P>Example for one node:</P><P>&nbsp;</P><P>hadoop has its own partition<BR />bash 'du -h --max-depth=1' in hadoop partition reports 'dn' directory is consuming 207G<BR />bash 'df -h' reports hadoop partition size 296G, used 208G,Avail 73G, Use% 75%</P><P>&nbsp;</P><P>Configured Capacity: 314825441690 (293.20 GB)&nbsp; -- good<BR />DFS Used: 221825508284 (206.59 GB)&nbsp; -- good<BR />Non DFS Used: 55394479116 (51.59 GB)&nbsp; -- ??? bash says 1G used outside of 'dn' directory in the partition<BR />DFS Remaining: 37605454290 (35.02 GB) -- ??? bash says 73G free<BR />DFS Used%: 70.46%<BR />DFS Remaining%: 11.94%</P><P>&nbsp;</P><P>fsck reports healthy</P><P>&nbsp;</P><P>redhat 6.9</P><P>5.8.2-1.cdh5.8.2.p0.3</P><P>&nbsp;</P><P>dfs.datanode.du.reserved == 1.96GiB<BR /><BR /></P><P>how to trouble-shoot ?</P><P>&nbsp;</P><P>thanks.</P><P>&nbsp;</P><P>hdfs dfsadmin -report</P><P>Configured Capacity: 1574127208450 (1.43 TB)<BR />Present Capacity: 1277963063885 (1.16 TB)<BR />DFS Remaining: 410632669242 (382.43 GB)<BR />DFS Used: 867330394643 (807.76 GB)<BR />DFS Used%: 67.87%<BR />Under replicated blocks: 0<BR />Blocks with corrupt replicas: 0<BR />Missing blocks: 0<BR />Missing blocks (with replication factor 1): 0</P><P>&nbsp;</P><P>hdfs fsck /</P><P>&nbsp;Total size:&nbsp;&nbsp;&nbsp; 281353009325 B<BR />&nbsp;Total dirs:&nbsp;&nbsp;&nbsp; 5236<BR />&nbsp;Total files:&nbsp;&nbsp; 501295<BR />&nbsp;Total symlinks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 (Files currently being written: 37)<BR />&nbsp;Total blocks (validated):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 501272 (avg. block size 561278 B)<BR />&nbsp;Minimally replicated blocks:&nbsp;&nbsp; 501272 (100.0 %)<BR />&nbsp;Over-replicated blocks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 (0.0 %)<BR />&nbsp;Under-replicated blocks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 (0.0 %)<BR />&nbsp;Mis-replicated blocks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 (0.0 %)<BR />&nbsp;Default replication factor:&nbsp;&nbsp;&nbsp; 3<BR />&nbsp;Average block replication:&nbsp;&nbsp;&nbsp;&nbsp; 3.0<BR />&nbsp;Corrupt blocks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0<BR />&nbsp;Missing replicas:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 (0.0 %)<BR />&nbsp;Number of data-nodes:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5<BR />&nbsp;Number of racks:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1</P> Thu, 07 Dec 2017 17:53:01 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Non-DFS-Used-is-reported-much-bigger-than-bash-df-h-reports/m-p/62569#M72457 ScottChris 2017-12-07T17:53:01Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Non-DFS-Used-is-reported-much-bigger-than-bash-df-h-reports/m-p/62624#M72458 <P>This problem is HDFS-9530 which has a fix in CDH-5.9.0.</P><P>&nbsp;</P><P>Bouncing the DN instances cleared this issue manually until we upgrade.</P><P>&nbsp;</P> Fri, 08 Dec 2017 22:19:12 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Non-DFS-Used-is-reported-much-bigger-than-bash-df-h-reports/m-p/62624#M72458 ScottChris 2017-12-08T22:19:12Z