Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

datanode stopped working

datanode stopped working

Expert Contributor

Everything in my cluster was working good and there were no warnings in ambari but today outofnowhere datanode stopped working.

ambari showed 0/1 live datanode ,I could successfully start it each time but ambari could never see it as live and soon it stopped.

I checked the logs and it showed this:

java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2016-05-05 16:46:58,282 ERROR datanode.DataNode (DataXceiver.java:run(278)) - warehouse.swtched.com:50010:DataXceiver error processing unknown operation  src: /10.10.10.9:60107 dst: /10.10.10.9:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)

How do I debug this?

I tried everything but had to reinstall stuff from scratch. It just does not look good or acceptable. What could be the way to debug such errors. where did it come from when there were no changes made anywhere in HDFS?

1 REPLY 1
Highlighted

Re: datanode stopped working

@sameer lail

Hi, There is an known bug which seems matching to your logs, however I don't think if these error messages can cause any issue. Can you please check the datanode log file for any other error exceptions?

https://issues.apache.org/jira/browse/AMBARI-12420

Don't have an account?
Coming from Hortonworks? Activate your account here