Support Questions
Find answers, ask questions, and share your expertise

DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010

Explorer

I am using hortonwork 2.3.0, having 10+ node cluster

1) One Datanode node is going down logs are :

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: datanode4:50010:DataXceiver error processing unknown operation src: /127.0.0.1:34350 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2016-11-21 04:37:51,386 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: hdp-n4:50010:Number of active connections is: 6 2016-11-21 04:37:51,395 DEBUG io.netty.channel.DefaultChannelPipeline: Discarded inbound message EmptyLastHttpContent that reached at the tail of the pipeline. Please check your pipeline configuration. 2016-11-21 04:37:52,167 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8010: task running 2016-11-21 04:38:02,168 DEBUG org.apache.hadoop.ipc.Server: IPC Server idle connection scanner for port 8010: task running

2) why source port is 34350 (is this is a random port taken by hdfs service )? as 50010 is for data transfer

3) IPC Server idle connection scanner for port 8010 ??

4) Interesting stuff :

a) ps -ef | grep datanode : Ans =39276

b) netstat -nap | grep 39276 Ans:

" tcp 1 0 127.0.0.1:35658 127.0.0.1:46869 CLOSE_WAIT 39276/java

unix 2 [ ACC ] STREAM LISTENING 317361 39276/java /var/lib/hadoop-hdfs/dn_socke

unix 2 [ ] STREAM CONNECTED 317761 39276/java

unix 3 [ ] STREAM CONNECTED 317365 39276/java

unix 3 [ ] STREAM CONNECTED 317364 39276/java

unix 2 [ ] STREAM CONNECTED 317359 39276/java "

when i am killing this process then trying to restart datanode from Ambari ,getting error " datanode is running on port and netstat on that port so many close wait connection.

If i restart the server port is getting free and able to start datanode for 24 hours after that same issue.

JPS :

7280 -- process information unavailable

6834 DAGAppMaster

5330 supervisor

9508 Jps

5636 logviewer

7222 -- process information unavailable

7721 DAGAppMaster

39276 DataNode

7167 -- process information unavailable

5071 NodeManager

1 REPLY 1

@Raj Kadel

ERROR datanode.DataNode (DataXceiver.java:run(278)) - DA-CLS-03-002:50010:DataXceiver error processing unknown operation src: /127.0.0.1:50842 dst: /127.0.0.1:50010

java.io.EOFException 
at java.io.DataInputStream.readShort(DataInputStream.java:315) 
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) 
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) 
at java.lang.Thread.run(Thread.java:745) 
ROOT CAUSE:

BUG-36881

These messages are caused by Ambari alerts monitoring the DataXceiver port on the datanode (port 50010) RESOLUTION: These messages can be safely ignored. Eventually,

the way Ambari monitors the datanodes will change so that these messages no longer occur. Link