question Re: Error while executing hdfs fsck command in Support Questions

Error while executing hdfs fsck command

wert_1311 — Fri, 16 Sep 2022 14:40:33 GMT

Hello,

I am trying to execute hdfs fsck command and am getting below error:

[root@server1 root]# hdfs fsck / > /home/test/fsck_output_2-11-21
Connecting to namenode via http://server1.com:50070/fsck?ugi=hdfs&path=%2F
Exception in thread "main" java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:363)
at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72)
at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:157)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:406)

Thanks

wert

Re: Error while executing hdfs fsck command

GangWar — Thu, 11 Feb 2021 11:22:49 GMT

@wert_1311 This could be a known issue with HDFS.

The issue with fsck command was introduced in HDFS-2538 when fsck doesn't show dots by default. fsck uses http port, 9871, which has the 60 seconds timeout hardcoded. The issue is observed when the command executed for more than the configured timeout.

Use -showprogress parameter that can help you.


# hdfs fsck / -showprogress

Re: Error while executing hdfs fsck command

wert_1311 — Mon, 15 Feb 2021 02:31:42 GMT

@GangWar Thanks !!! that works

Re: Error while executing hdfs fsck command

USMAN_HAIDER — Mon, 26 Jul 2021 12:09:00 GMT

Hey @GangWar Thank you for the solution but when I use this comman:

hdfs fsck / -showprogress

It shows all the corrupted blocks but not delete these block I check the blocks status form my CM UI its still the same. I run this command for 2 hours but none of missing block is delete.
When I use this Command:

hdfs fsck / | egrep -v '^\.+$' | grep -v replica

hdfs fsck / -delete

Its give the same error that highlighted by @wert_1311 in the above post. First time this commands works and it delete some blocks but after some time I am facing this issue while running this command.

Here I am attaching the snapshot

Re: Error while executing hdfs fsck command

GangWar — Fri, 30 Jul 2021 13:58:49 GMT

This seems a Connection time out to the name node so you may want to check if there is any Network issue or something else going on with CM's help. That should give you some clue.