dfs storage(dfs.data.dir) space issue

[#1]

Hi,

My Cluster has 3 nodes which occupies almost 870 GB. 

[hdfs@XXXX bin]$ hadoop fs -du -s -h /user/hdfs
435.3 G 870.7 G /user/hdfs

However the space held by dfs.data.dir directory on single node is more than total space occupied by the cluster.

[hdfs@XXXX bigdata]$ du -sh dfs
464G dfs

[hdfs@YYYY bin]$ du -sh /bigdata/dfs
746G /bigdata/dfs

[hdfs@ZZZZ ~]$ du -sh /bigdata/dfs
257G /bigdata/dfs

Isn't hdfs files are stored on dfs.data.dir directory so how the space occupied is more than that of cluster?

Please help me reduce the space of dfs directory as it is in critical state.

[hdfs@XXXX finalized]$ ls -lart
total 2008
drwxr-xr-x 244 hdfs hdfs 12288 Mar  7  2017 subdir0
drwxr-xr-x 258 hdfs hdfs 12288 Mar  7  2017 subdir1
drwxr-xr-x 258 hdfs hdfs 12288 Mar 10  2017 subdir2
drwxr-xr-x 258 hdfs hdfs 12288 Mar 12  2017 subdir3
drwxr-xr-x 258 hdfs hdfs 12288 Mar 14  2017 subdir4
drwxr-xr-x 258 hdfs hdfs 12288 Mar 15  2017 subdir5
drwxr-xr-x 258 hdfs hdfs 12288 Mar 15  2017 subdir6
drwxr-xr-x 258 hdfs hdfs 12288 Mar 16  2017 subdir7
drwxr-xr-x 258 hdfs hdfs 12288 Mar 17  2017 subdir8
drwxr-xr-x 258 hdfs hdfs 12288 Mar 17  2017 subdir9
drwxr-xr-x 258 hdfs hdfs 12288 Mar 17  2017 subdir10
.
.
.
drwxr-xr-x 258 hdfs hadoop 12288 Nov 14 03:16 subdir119
drwxr-xr-x 258 hdfs hadoop 12288 Nov 14 05:04 subdir122
drwxr-xr-x 5 hdfs hadoop 4096 Nov 14 14:25 subdir181


[hdfs@XXXX finalized]$ pwd
/bigdata/dfs/dn/current/BP-939287337-10.0.0.4-1484085163925/current/finalized
[hdfs@XXXX finalized]$

These subdir have more subdirs under it and finally blocks. Can I delete the older ones?

Thanks,

Shilpa

[#2 · Accepted solution]

I copied the older subdir* from dfs/dn/current/BP<...>/current/finalized to another storage mount/drive. This has not affected my data on hdfs.

Please let me know if someone thinks there is a better way than this workaround.

Thanks,

Shilpa