Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Delete NON DFS used space in HDFS filesystem

Delete NON DFS used space in HDFS filesystem

I have allocated 4GB as Reserved Space for Non DFS used (dfs.datanode.du.reserved) and also configured separate disk partitions for Non HDFS use like Intermediate data. 

 As per the "hdfs dfsadmin -report", I see that my "Non DFS used" in my YARN cluster is growing more than the DFS used, please suggest how to delete the "Non DFS used"  so that I can increase the disk space for my HDFS data directories?

[hduser@node2 ~]$ sudo -u hdfs hdfs dfsadmin -report
[sudo] password for hduser:
Configured Capacity: 47518140008 (44.25 GB)
Present Capacity: 46712834138 (43.50 GB)
DFS Remaining: 45923338126 (42.77 GB)
DFS Used: 789496012 (752.92 MB)
DFS Used%: 1.69%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.52.111:50010 (node1.example.com)
Hostname: node1.example.com
Rack: /default
Decommission Status : Normal
Configured Capacity: 23759070004 (22.13 GB)
DFS Used: 394748006 (376.46 MB)
Non DFS Used: 402652935 (384.00 MB)
DFS Remaining: 22961669063 (21.38 GB)
DFS Used%: 1.66%
DFS Remaining%: 96.64%
Configured Cache Capacity: 121634816 (116 MB)
Cache Used: 0 (0 B)
Cache Remaining: 121634816 (116 MB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Mon May 23 20:52:55 IST 2016


Name: 192.168.52.112:50010 (node2.example.com)
Hostname: node2.example.com
Rack: /default
Decommission Status : Normal
Configured Capacity: 23759070004 (22.13 GB)
DFS Used: 394748006 (376.46 MB)
Non DFS Used: 402652935 (384.00 MB)
DFS Remaining: 22961669063 (21.38 GB)
DFS Used%: 1.66%
DFS Remaining%: 96.64%
Configured Cache Capacity: 523239424 (499 MB)
Cache Used: 0 (0 B)
Cache Remaining: 523239424 (499 MB)
Cache Used%: 0.00%
Cache Remaining%: 100.00%
Xceivers: 8
Last contact: Mon May 23 20:52:56 IST 2016

 

1 REPLY 1

Re: Delete NON DFS used space in HDFS filesystem

Master Guru
I don't see the Non DFS Used as too high (400~ MB per DN in your report output).

In any case, to delete the Non-DFS Used Space you will first need to identify what counts as part of it. Typically any space usage OUTSIDE of the DataNode disk directories, on the same disk mount, is counted as Non-DFS Used Space (its calculated as Capacity - DFS Used - Free). So inspect your mount points to identify what directory other than the DN ones are consuming space on the same disk and then decide if you can delete them.
Don't have an account?
Coming from Hortonworks? Activate your account here