Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Non DFS Used is reported much bigger than bash 'df -h' reports, reducing DFS remaining

SOLVED Go to solution

Non DFS Used is reported much bigger than bash 'df -h' reports, reducing DFS remaining

Contributor

Seeing this issue on all data nodes.

 

Example for one node:

 

hadoop has its own partition
bash 'du -h --max-depth=1' in hadoop partition reports 'dn' directory is consuming 207G
bash 'df -h' reports hadoop partition size 296G, used 208G,Avail 73G, Use% 75%

 

Configured Capacity: 314825441690 (293.20 GB)  -- good
DFS Used: 221825508284 (206.59 GB)  -- good
Non DFS Used: 55394479116 (51.59 GB)  -- ??? bash says 1G used outside of 'dn' directory in the partition
DFS Remaining: 37605454290 (35.02 GB) -- ??? bash says 73G free
DFS Used%: 70.46%
DFS Remaining%: 11.94%

 

fsck reports healthy

 

redhat 6.9

5.8.2-1.cdh5.8.2.p0.3

 

dfs.datanode.du.reserved == 1.96GiB

how to trouble-shoot ?

 

thanks.

 

hdfs dfsadmin -report

Configured Capacity: 1574127208450 (1.43 TB)
Present Capacity: 1277963063885 (1.16 TB)
DFS Remaining: 410632669242 (382.43 GB)
DFS Used: 867330394643 (807.76 GB)
DFS Used%: 67.87%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

 

hdfs fsck /

 Total size:    281353009325 B
 Total dirs:    5236
 Total files:   501295
 Total symlinks:                0 (Files currently being written: 37)
 Total blocks (validated):      501272 (avg. block size 561278 B)
 Minimally replicated blocks:   501272 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          5
 Number of racks:               1

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Non DFS Used is reported much bigger than bash 'df -h' reports, reducing DFS remaining

Contributor

This problem is HDFS-9530 which has a fix in CDH-5.9.0.

 

Bouncing the DN instances cleared this issue manually until we upgrade.

 

1 REPLY 1
Highlighted

Re: Non DFS Used is reported much bigger than bash 'df -h' reports, reducing DFS remaining

Contributor

This problem is HDFS-9530 which has a fix in CDH-5.9.0.

 

Bouncing the DN instances cleared this issue manually until we upgrade.