Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010

Highlighted

DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010

Explorer

I am using hortonwork 2.3.0, having 10+ node cluster

1) One Datanode node is going down logs are :

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: datanode4:50010:DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2016-11-21 04:37:51,386 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: hdp-n4:50010:Number of active connections is: 6 2016-11-21 04:37:51,395 DEBUG io.netty.channel.DefaultChannelPipeline: Discarded inbound message EmptyLastHttpContent that reached at the tail of the pipeline. Please check your pipeline configuration. 2016-11-21 04:37:52,167 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8010: task running 2016-11-21 04:38:02,168 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8010: task running

2) why source port is 34350 (is this is a random port taken by hdfs service )? as 50010 is for data transfer

3) IPC Server idle connection scanner for port 8010 ??

4) Interesting stuff :

a) ps -ef | grep datanode : Ans =39276

b) netstat -nap | grep 39276 Ans:

" tcp 1 0 127.0.0.1:35658 127.0.0.1:46869 CLOSE_WAIT 39276/java

unix 2 [ ACC ] STREAM LISTENING 317361 39276/java /var/lib/hadoop-hdfs/dn_socke

unix 2 [ ] STREAM CONNECTED 317761 39276/java

unix 3 [ ] STREAM CONNECTED 317365 39276/java

unix 3 [ ] STREAM CONNECTED 317364 39276/java

unix 2 [ ] STREAM CONNECTED 317359 39276/java "

when i am killing this process then trying to restart datanode from Ambari ,getting error " datanode is running on port and netstat on that port so many close wait connection.

If i restart the server port is getting free and able to start datanode for 24 hours after that same issue.

JPS :

7280 -- process information unavailable

6834 DAGAppMaster

5330 supervisor

9508 Jps

5636 logviewer

7222 -- process information unavailable

7721 DAGAppMaster

39276 DataNode

7167 -- process information unavailable

5071 NodeManager

1 REPLY 1
Highlighted

Re: DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010

@Raj Kadel

ERROR datanode.DataNode (DataXceiver.java:run(278)) - DA-CLS-03-002:50010:DataXceiver error processing unknown operation src: /127.0.0.1:50842 dst: /127.0.0.1:50010

java.io.EOFException 
at java.io.DataInputStream.readShort(DataInputStream.java:315) 
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) 
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) 
at java.lang.Thread.run(Thread.java:745) 
ROOT CAUSE:

BUG-36881

These messages are caused by Ambari alerts monitoring the DataXceiver port on the datanode (port 50010) RESOLUTION: These messages can be safely ignored. Eventually,

the way Ambari monitors the datanodes will change so that these messages no longer occur. Link

Don't have an account?
Coming from Hortonworks? Activate your account here