Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there a command to find the number of blocks pending deletion in HDFS?

SOLVED Go to solution
Highlighted

Is there a command to find the number of blocks pending deletion in HDFS?

Contributor

I am currently deleting a large amount of small files and am needing to monitor the number of blocks pending deletion so I do not put too much of a load on the cluster. I know this number can be viewed in the NameNode UI, but I would like to run a command instead which I can then use in combination with `watch` so I can get continuous updates. Does such a command exist?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Hi @Josh Nicholson

This works for me, "PendingDeletionBlocks" is what you're looking for in the output if you're looking for pending deletes only.

watch curl --silent http://<your-name-node>:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
6 REPLIES 6

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Hi @Josh Nicholson

This works for me, "PendingDeletionBlocks" is what you're looking for in the output if you're looking for pending deletes only.

watch curl --silent http://<your-name-node>:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Contributor

I didn't think of querying the JMX, this works fantastically, thanks!

Actually @Jonathan Sneep I am not able to curl to the JMX as I need authorization. This cluster is kerberized and using SSL. I tried searching around on how to pass credentials but couldn't find anything. Any ideas?

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Hey @Josh Nicholson
curl --negotiate -u should work, I will test this myself in a bit.

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Contributor

That worked, thanks.

Re: Is there a command to find the number of blocks pending deletion in HDFS?

Perfect, glad to hear!

Re: Is there a command to find the number of blocks pending deletion in HDFS?

New Contributor

goto Ambari --> HDFS --> QuickLinks --> Master(Active) ... a new page will open ... search "Number of Blocks Pending Deletion"... you will find the pending deletion blocks.. refresh page after 30 sec. counts will be changed... enjoy