question Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!! in Support Questions

Cannot start data node - did the troubleshooting and find something wired: could you help!!!

luc_tiber — Sun, 31 Jan 2016 17:19:51 GMT

I installed Hadoop 2.7.2 (1 master NN -1 second NN-3 datanodes) and tried to start my datanodes !!! Get stuck!

After trouble shouting the logs (see below), the fatal error is due to ClusterID mismatch... easy! just change the IDs. WRONG... when I checked my VERSION files on the NameNode and the DataNodes they are identical.(see by yourself below)

So the question is simple: INTO the log file --> Where the ClusterID of the NameNode is coming From????

LOG FILE:

WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException:Incompatible clusterIDs in/home/hduser/mydata/hdfs/datanode: namenode clusterID =**CID-8e09ff25-80fb-4834-878b-f23b3deb62d0**; datanode clusterID =**CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1**
org.apache.hadoop.hdfs.server.datanode.DataNode:Initialization failed forBlock pool <registering>(DatanodeUuid unassigned) service to master/172.XX.XX.XX:9000.Exiting.
java.io.IOException:All specified directories are failed to load.
atorg.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
atorg.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358)
atorg.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323)
atorg.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
atorg.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
atorg.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:745)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Ending block pool service for:Block pool <registering>(DatanodeUuid unassigned) service to master/172.XX.XX.XX:9000
INFO org.apache.hadoop.hdfs.server.datanode.DataNode:RemovedBlock pool <registering>(DatanodeUuid unassigned)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode:ExitingDatanode

COPY of THE VERSION FILE

the master

storageID=DS-f72f5710-a869-489d-9f52-40dadc659937
clusterID=CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1
cTime=0
datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b
storageType=DATA_NODE
layoutVersion=-56

THE DataNode

storageID=DS-f72f5710-a869-489d-9f52-40dadc659937
clusterID=CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1
cTime=0
datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b
storageType=DATA_NODE
layoutVersion=-56

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

nsabharwal — Sun, 31 Jan 2016 19:33:56 GMT

@luc tiber

I am guessing that it's non ambari install. Good thread

namenode clusterID =**CID-8e09ff25-80fb-4834-878b-f23b3deb62d0**;

datanode clusterID =**CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1**

You can see this for detail on version file location

Resolution: As it's new cluster so we can reformat the namenode as discussed here thread

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

aervits — Sun, 31 Jan 2016 20:23:06 GMT

@luc tiber Hadoop 2.7.2 is not released by Hortonworks yet. Mixing a HWX release with any other including Apache will yeld unintended results

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

luc_tiber — Sun, 31 Jan 2016 22:46:31 GMT

Hi Artem,

I appreciate Hortonwork perspective on the issue I shared. This is a clear product management position and from that perspective, this is a very valid point. Point taken!

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

nsabharwal — Sun, 31 Jan 2016 23:36:22 GMT

@luc tiber I agree with your last comments and glad that you are trying latest version. Please submit your feedback once you fix the issue.

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

luc_tiber — Mon, 01 Feb 2016 02:16:57 GMT

Just to summarize (and close) this issue, I would like to share how I fixed this issue.

On the MASTER and the 2nd Namenode the Namenode VERSION file is under ~/.../namenode/current/VERSION.

BUT for DATANODES the path is different. it should look something like this ~/.../datanode/current/VERSION

ClusterIDs between the 2 VERSION files should be identical

Hope it helps!

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

nsabharwal — Mon, 01 Feb 2016 02:31:30 GMT

@luc tiber

Thanks for sharing the update !!! I may reproduce and test this

Re: Cannot start data node - did the troubleshooting and find something wired: could you help!!!

HKG — Mon, 10 Feb 2020 16:47:23 GMT

Awesome.It worked for me.