Skip to main content
hanumanth
Gold ⭐️
November 4, 2022
Solved

Data directories mount point space not reclaimed

Hi Team,

 

We have 3 DN's and Data Directories mount point size is around 1TB on each Node , total Data Directories size was 3TB. I deleted 500GB data from HDFS but space got release from only DN3 Data Directory not from other DN's.

 

Please advise..

 

blizano
blizanoAnswer
Gold ⭐️
November 14, 2022

@hanumanth you may check if the files you deleted from HDFS still exist somehow in HDFS and if so, check the replication factor applied to them, this can be contained in the second column of the hdfs -ls output, for this you can collect a recursive listing by running

 

hdfs dfs -ls -R / > hdfs_recursive

 

After you can try a filter to this file to know the RF factor applied to your files: 

 

hdfs dfs -ls -R / | awk '{print $2}' | sort | uniq -c

 

Also, ensure that there is no other content from other processes (something different than HDFS) filling up the mount points used to store the HDFS blocks.

 

You can get also the following outputs to compare if existing data is protected by snapshots or not:

 

#1 du -s -x

#2 du -s

 

---> Du command usage [1] 

 

If the results of the above commands differ between them probably there are snapshots still present that are avoiding blocks to be deleted from datanodes. 

 

6 replies

pajoshi
Gold ⭐️
Gold ⭐️
November 4, 2022

Hi @hanumanth,

 

Could you please check below things:

1) If the replication factor of deleted files was 1.

2) If there are blocks still pending to be deleted. This could be checked from NN UI

3) If there are hdfs snapshots configured on the deleted paths or its parent directory

 

Thank you

Translate
hanumanth
hanumanthAuthor
Gold ⭐️
November 14, 2022

Hi Pajoshi,

 

Please find my inline comments.

 

1) If the replication factor of deleted files was 1.

 

-->Replication factor - 2

2) If there are blocks still pending to be deleted. This could be checked from NN UI

->we observed 6 blocks pending to be deleted.

3) If there are hdfs snapshots configured on the deleted paths or its parent directory

--> We deleted hdfs snapshots

 

Translate
DianaTorres
Community Manager
Community Manager
November 7, 2022

@hanumanth Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Translate
Regards, Diana Torres, Senior Community Moderator
hanumanth
hanumanthAuthor
Gold ⭐️
November 14, 2022

not yet issue didn't resolved

Translate
blizano
Gold ⭐️
blizanoAnswer
Gold ⭐️
November 14, 2022

@hanumanth you may check if the files you deleted from HDFS still exist somehow in HDFS and if so, check the replication factor applied to them, this can be contained in the second column of the hdfs -ls output, for this you can collect a recursive listing by running

 

hdfs dfs -ls -R / > hdfs_recursive

 

After you can try a filter to this file to know the RF factor applied to your files: 

 

hdfs dfs -ls -R / | awk '{print $2}' | sort | uniq -c

 

Also, ensure that there is no other content from other processes (something different than HDFS) filling up the mount points used to store the HDFS blocks.

 

You can get also the following outputs to compare if existing data is protected by snapshots or not:

 

#1 du -s -x

#2 du -s

 

---> Du command usage [1] 

 

If the results of the above commands differ between them probably there are snapshots still present that are avoiding blocks to be deleted from datanodes. 

 

Translate
DianaTorres
Community Manager
Community Manager
November 17, 2022

@hanumanth Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Translate
Regards, Diana Torres, Senior Community Moderator