Datanode is not coming up, failing with Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNode

[#1]

Was an interesting issue faced last week. Putting here for bigger audience, might be helpful to others too.

PROBLEM

On one of the node, datanode and nodemanager were not coming up. Below is the error after starting from ambari.

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-ny-node3.hwxblr.com.out
Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNode

As datanode process itself wasn't loaded, so nothing was printed in datanode logs. Only thing we see in .out file is

Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNode

Verified in the jar, DataNode class was present.

/usr/jdk64/jdk1.8.0_77/bin/jar  -tvf /usr/hdp/2.5.0.0-1245/hadoop-hdfs/hadoop-hdfs-2.7.3.2.5.0.0-1245.jar | grep DataNode.class 

org/apache/hadoop/hdfs/server/datanode/DataNode.class

ROOT CAUSE

@nvadivelu came to rescue. We used below utility to figure out which class was missing.

public class Sample {
public static void main(String[] args) {
try {
org.apache.hadoop.hdfs.server.datanode.DataNode.main(args);
} catch (Throwable ex) {
ex.printStackTrace();
}
}
}

We ran the above code, and it printed the exact class which wasn't able to load.

/usr/jdk64/jdk1.8.0_77/bin/javac -cp `hadoop classpath` Sample.java
Sample.java:5: error: cannot access TraceAdminProtocol
org.apache.hadoop.hdfs.server.datanode.DataNode.main(args);
                                               ^
  class file for org.apache.hadoop.tracing.TraceAdminProtocol not found
1 error

TraceAdminProtocol clas is present hadoop-common jar. We grep this class in the hadoop-common jar, we didn't find. But on other host, where datanode was running fine, we got below results.

grep "TraceAdminProtocol" /usr/hdp/2.5.0.0-1245/hadoop/hadoop-common-2.7.3.2.5.0.0-1245.jar 

Binary file /usr/hdp/2.5.0.0-1245/hadoop/hadoop-common-2.7.3.2.5.0.0-1245.jar matches

Also we verified size of this jar was less compared to the working one.

RESOLUTION

We copied this jar from the working host and datanode and nodemanager came up fine. We had no clue, from where this jar came, even of same version. But it was a good learning experience.