Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDFS block count does not decrease after deleting data

avatar
Contributor

Hi,

after having deleted tera bytes of data from HDFS (1/4 of the total capacity), the block count among data nodes did not decrease as expected. It is still over the critical threshold.

How could it be solved?

Thank you

 

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
13 REPLIES 13

avatar
Contributor

Hi,

I'm still analyzing the output: the command "fsck" on the path where deleting operations have been made reports just 1 block. 

Looking at the attached chart, you can see that on May, the 19th, a lot of data was removed from hdfs (60TB), and the number of blocks decreased for a single datanode (bda1node02). 

600.000 blocks (1 block -> 256MB).
In the other datanodes, blocks remained the same (or increased slightly). 


blocks.PNG

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Hi @mszurap ,

I agree with you about these numbers. 

Even if 60-100TB is a high amount of data, the total number of blocks involved is not so high (next to 600k), if compared to each Datanode. 

 

Each datanode reports 9M of blocks, but we found the problem is related to other directories that cointain small files, where block size is about 2-3MB. Even if the total size of these directory is not so high, we expect the number of block will decrease more significantly.

 

We are facing the problem of small files, which determines a high number of blocks. The directory we have deleted had larger blocks, which is why the decrease in blocks was imperceptible.

 

Thank you for the support in the analysis!

 

avatar

Hi Andrea,

Great to see that it has been found now and thanks for marking the post as answered.

All the best, Miklos