question files/directories occupying maximum space in hdfs in Archives of Support Questions (Read Only)

files/directories occupying maximum space in hdfs

arunpoy — Thu, 21 Jul 2016 12:16:52 GMT

I need to find the files/directories that are occupying the maximum space in HDFS. so that i can ask the owners of those directories to clean up the spaces. what is the command that can provide the results for this purpose. Also i do notice the namenode heap size is increasing and is around 80 - 90%. IS there a way to find which files cause this . i.e which directory has too many small files or how to identify the small files which might cause this .

Or is there any other reason thaat is causing the namenode heapsize to increase.

My name node memory is 3 GB.

Re: files/directories occupying maximum space in hdfs

sbhat — Thu, 21 Jul 2016 13:04:26 GMT

@ARUN

Try the command

hadoop fs -du -h /

This should get us the space occupied by directories in hdfs. To drill down, change / to the directory you want to check.

Re: files/directories occupying maximum space in hdfs

divakarreddy_a — Thu, 21 Jul 2016 13:10:30 GMT

@ARUN

1) Identifying directories usage in HDFS

Try to run below for each directory to know the usage

[root@abc01 ~]# hadoop fs -du -s -h <hdfs location>

50.9 M /abc

[root@adc01 ~]#

2) namenode heap size is increasing and is around 80 - 90%.

NN heap size increasing because of your cluster is occupying maximum space in HDFS for files & directories.

Here is the link for increaseing values based on file size

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html

I hope this will help you.