Created 05-23-2016 03:31 PM
I have allocated 4GB as Reserved Space for Non DFS used (dfs.datanode.du.reserved) and also configured separate disk partitions for Non HDFS use like Intermediate data. As per the "hdfs dfsadmin -report", I see that my "Non DFS used" in my YARN cluster is growing more than the DFS used, please suggest how to delete the "Non DFS used" so that I can increase the disk space for my HDFS data directories?
Created 05-23-2016 03:40 PM
You can check if your yarn-local and yarn-logs directories for contents. As long as the job is successful, these get cleared but there can be cases during jobs failures where they are not purged.
Created 05-23-2016 03:44 PM
Non DFS is not related to HDFS storage, these are the local files/directories processes logs or tmp log/dir in some local node partition. Try deleting logs from your local nodes and see if that make some space, look for yarn.nodemanager.local-dirs parameter location.