Hide "Connecting to namenode via..." message from stdout when using hdfs fsck in a for loop

juan_manuel_nie — Fri, 25 Nov 2016 18:33:45 GMT

Hello ,

I usually use for loops to get info of some folders.

For example: I had to find which folder inside /user/bigdata was consuming high number of blocks due to small files.

So i used this:

for i in $(hadoop fs -ls /user/bigdata/ | grep drwx | awk '{print $8}'); do echo "$i $(hdfs fsck $i -blocks -files -locations | grep BP- | wc -l)" ; done

Getting a lot of "Connecting to namenode via http://<hostname>:50070/fsck?ugi=hdfs&blocks=1&files=1&locations=1&path=%2Fuser%2Fbigdata%2F.<directory>" messages.

Does exist any way of hide this message? I currently have to redirect the output to a file and then use cat to read a clear infomation.

Thank you in advance.

Re: Hide "Connecting to namenode via..." message from stdout when using hdfs fsck in a for loop

juan_manuel_nie — Fri, 25 Nov 2016 19:42:04 GMT

Well finally I solved this.

The "Connecting to namenode via http://<hostname>:50070/fsck?ugi=hdfs█s=1&files=1&locations=1&path=%2Fuser%2Fbigdata%2F.<directory>" is the stderr output of the command so redirecting stderr to /dev/null does the work :).

for i in $(hadoop fs -ls /user/bigdata/ | grep drwx | awk '{print $8}'); do echo "$i $(hdfs fsck $i -blocks -files -locations 2> /dev/null | grep BP- | wc -l)" ; done

question Hide "Connecting to namenode via..." message from stdout when using hdfs fsck in a for loop in Archives of Support Questions (Read Only)

Hide "Connecting to namenode via..." message from stdout when using hdfs fsck in a for loop

Re: Hide "Connecting to namenode via..." message from stdout when using hdfs fsck in a for loop