Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to delete "NON DFS used" space in HDFS filesystem

How to delete "NON DFS used" space in HDFS filesystem

I have allocated 4GB as Reserved Space for Non DFS used (dfs.datanode.du.reserved) and also configured separate disk partitions for Non HDFS use like Intermediate data. As per the "hdfs dfsadmin -report", I see that my "Non DFS used" in my YARN cluster is growing more than the DFS used, please suggest how to delete the "Non DFS used" so that I can increase the disk space for my HDFS data directories?

2 REPLIES 2

Re: How to delete "NON DFS used" space in HDFS filesystem

Guru

You can check if your yarn-local and yarn-logs directories for contents. As long as the job is successful, these get cleared but there can be cases during jobs failures where they are not purged.

Highlighted

Re: How to delete "NON DFS used" space in HDFS filesystem

@Avinash C

Non DFS is not related to HDFS storage, these are the local files/directories processes logs or tmp log/dir in some local node partition. Try deleting logs from your local nodes and see if that make some space, look for yarn.nodemanager.local-dirs parameter location.

Don't have an account?
Coming from Hortonworks? Activate your account here