Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hdfs fsck / breaks with IOException: Premature EOF

Highlighted

hdfs fsck / breaks with IOException: Premature EOF

New Contributor

Hi there,

I am running hdfs fsck / on a cluster with HDP 2.4.2. Unfortunately, it breaks with an "IOException: Premature EOF":

$ hadoop version
Hadoop 2.7.1.2.4.2.10-1
Subversion git@github.com:hortonworks/hadoop.git -r 183deee535d7360e487dccb66071a10995a304ac
Compiled by jenkins on 2016-07-12T20:29Z
Compiled with protoc 2.5.0
From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
This command was run using /usr/hdp/2.4.2.10-1/hadoop/hadoop-common-2.7.1.2.4.2.10-1.jar

$ sudo -u hdfs hdfs fsck /
[...]
..................
Exception in thread "main" java.io.IOException: Premature EOF
        at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
        at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
        at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3336)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:346)
        at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:73)
        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:151)
        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:148)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
        at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:379)

Any hints would be highly appreciated.

Best Michael

7 REPLIES 7

Re: hdfs fsck / breaks with IOException: Premature EOF

Super Guru
@Michael Häusler

You are running into a security issue. This seems like a Kerberized cluster and you probably need to do a kinit first before you can run the command. If you have done kinit, then the user you did kinit with appears to not have permission to impersonate user "hdfs".

Re: hdfs fsck / breaks with IOException: Premature EOF

New Contributor

@mqureshi

Thanks a lot for your advice.

It might not be security related. Impersonation works just fine when checking for a subset of the filesystem, e.g.:

$ sudo -u hdfs hdfs fsck /apps
Connecting to namenode via http://********:50070/fsck?ugi=hdfs&path=%2Fapps
FSCK started by hdfs (auth:SIMPLE) from /***.***.***.*** for path /apps at Fri Nov 25 19:14:54 UTC 2016
..Status: HEALTHY
 Total size:    269336194 B
 Total dirs:    6
 Total files:   2
 Total symlinks:                0
 Total blocks (validated):      3 (avg. block size 89778731 B)
 Minimally replicated blocks:   3 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          40
 Number of racks:               4
FSCK ended at Fri Nov 25 19:14:54 UTC 2016 in 0 milliseconds




The filesystem under path '/apps' is HEALTHY

But a check on the full HDFS fails with Premature EOF. Any other ideas?

Re: hdfs fsck / breaks with IOException: Premature EOF

Super Guru

@Michael Häusler

That's probably because the user you did a kinit with has access to "/apps" but not to "/". I am also surprised by "auth:Simple". If your cluster is Kerberized then auth:Simple should not work. Have you done a kinit?

If you look at your stacktrace, it has the following exceptions which indicates a Kerberos/Permission issue

 at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:422)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)

Re: hdfs fsck / breaks with IOException: Premature EOF

New Contributor

@mqureshi

Thanks for your reply. The cluster is not kerberized. I am running fsck as user "hdfs", which owns both directories.

The stacktrace seems to indicate that doPrivileged called DFSck$1.run. So impersonation should have been successful.

Re: hdfs fsck / breaks with IOException: Premature EOF

Super Guru

@Michael Häusler

Can you please log onto the linux box where namenode is running and check namenode logs? I think there might be more information there to help us debug.

Re: hdfs fsck / breaks with IOException: Premature EOF

New Contributor

@mqureshi Thanks for your reply. I checked the namenode logs. Unfortunately I see no messages which seem to relate to the fsck. Also, there are no WARN or ERROR messages during the timeframe of the fsck. The only message that seems correlated is:

2016-11-28 16:55:49,198 INFO  namenode.NameNode (NamenodeFsck.java:fsck(327)) - FSCK started by hdfs (auth:SIMPLE) from /***.***.***.*** for path / at Mon Nov 28 16:55:49 UTC 2016

Also restarting the namenode did not improve the situation.

Do you think it could be related to https://issues.apache.org/jira/browse/HDFS-7175 ?

Best regards

Michael

Re: hdfs fsck / breaks with IOException: Premature EOF

Super Guru

I don't think it's this issue because we don't see socket timeout in your stack trace. I have a tedious suggestion if you don't mind trying. It might help us narrow down our cause. Like you ran fsck on /app folder, if you don't have many folders in your root, then can you try running fsck on each of these one by one?

Don't have an account?
Coming from Hortonworks? Activate your account here