Seeing this issue on all data nodes.
Example for one node:
hadoop has its own partition
bash 'du -h --max-depth=1' in hadoop partition reports 'dn' directory is consuming 207G
bash 'df -h' reports hadoop partition size 296G, used 208G,Avail 73G, Use% 75%
Configured Capacity: 314825441690 (293.20 GB) -- good
DFS Used: 221825508284 (206.59 GB) -- good
Non DFS Used: 55394479116 (51.59 GB) -- ??? bash says 1G used outside of 'dn' directory in the partition
DFS Remaining: 37605454290 (35.02 GB) -- ??? bash says 73G free
DFS Used%: 70.46%
DFS Remaining%: 11.94%
fsck reports healthy
redhat 6.9
5.8.2-1.cdh5.8.2.p0.3
dfs.datanode.du.reserved == 1.96GiB
how to trouble-shoot ?
thanks.
hdfs dfsadmin -report
Configured Capacity: 1574127208450 (1.43 TB)
Present Capacity: 1277963063885 (1.16 TB)
DFS Remaining: 410632669242 (382.43 GB)
DFS Used: 867330394643 (807.76 GB)
DFS Used%: 67.87%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
hdfs fsck /
Total size: 281353009325 B
Total dirs: 5236
Total files: 501295
Total symlinks: 0 (Files currently being written: 37)
Total blocks (validated): 501272 (avg. block size 561278 B)
Minimally replicated blocks: 501272 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 5
Number of racks: 1
