I have allocated 4GB as Reserved Space for Non DFS used (dfs.datanode.du.reserved) and also configured separate disk partitions for Non HDFS use like Intermediate data. As per the "hdfs dfsadmin -report", I see that my "Non DFS used" in my YARN cluster is growing more than the DFS used, please suggest how to delete the "Non DFS used" so that I can increase the disk space for my HDFS data directories?
You can check if your yarn-local and yarn-logs directories for contents. As long as the job is successful, these get cleared but there can be cases during jobs failures where they are not purged.
Non DFS is not related to HDFS storage, these are the local files/directories processes logs or tmp log/dir in some local node partition. Try deleting logs from your local nodes and see if that make some space, look for yarn.nodemanager.local-dirs parameter location.