Support Questions
Find answers, ask questions, and share your expertise

Why there some metrics display no data available in HDFS and YARN components?

Explorer

I had build HDP 2.4 in the past year on product environment, and it run well many times. But now it had some mistake happened. The all components running is OK, but the metrics display is not good. So is strange it only trouble with HDFS and YARN components. I had shot the dashboard screen like below:

hdfs-err01.png

yarn-err01.png

Then also I had try find the detail info in log files but not success. Because there no any ERROR info in `ambari-metrics-collector.log`, So what I do fix this problems? Thanks.

40021-collector-err01.png

40019-yarn-err01.png

40018-hdfs-err01.png

2 REPLIES 2

Re: Why there some metrics display no data available in HDFS and YARN components?

Expert Contributor

@elkan li

What Ambari version is this?

Can you attach HDFS Namenode / Datanode logs?

Re: Why there some metrics display no data available in HDFS and YARN components?

Explorer

1. Apache Ambari Version is 2.1.2.1

2. Data Node Logs: (only catch the ERROR level info from file like hadoop-hdfs-datanode-xxxx.log)

2017-10-31 01:52:33,414 ERROR datanode.DataNode (DataXceiver.java:run(278)) - data-01:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:55901 dst: /127.0.0.1:50010

java.io.EOFException

        at java.io.DataInputStream.readShort(DataInputStream.java:315)

        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

        at java.lang.Thread.run(Thread.java:745)
2017-10-31 04:43:54,341 ERROR datanode.DataNode (DataXceiver.java:run(278)) - data-02:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:35030 dst: /127.0.0.1:50010

java.io.EOFException

        at java.io.DataInputStream.readShort(DataInputStream.java:315)

        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

        at java.lang.Thread.run(Thread.java:745)

2017-10-31 04:43:54,351 INFO  DataNode.clienttrace (DataXceiver.java:requestShortCircuitFds(369)) - src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: 1090201179, srvID: 37c8941a-1524-4526-adf3-7265b6013c06, success: true

2017-10-30 21:20:02,607 ERROR datanode.DataNode (DataXceiver.java:run(278)) - data-03:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:35388 dst: /127.0.0.1:50010

java.io.EOFException

        at java.io.DataInputStream.readShort(DataInputStream.java:315)

        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

        at java.lang.Thread.run(Thread.java:745)

2017-10-30 21:20:02,611 INFO  DataNode.clienttrace (DataXceiver.java:requestShortCircuitFds(369)) - src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: 1088782115, srvID: 8945905f-8ee7-4373-8115-d1c7a17574f5, success: true

There only can find this type error tip, So what's you mean for check this? Is still need attach other info?

Thanks.