Reply
Highlighted
Contributor
Posts: 28
Registered: ‎03-04-2014

Datanode issue after CDH5.2 upgrade

All,
 
 
I tried to upgrade my cluster from 5.0.1 to 5.2 following the documentation http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_upgrade_to_cdh52_...
 
Everything went fine , except the datanode when starting it , i got the exception below.
 
2014-12-15 15:28:53,734 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading block pool storage directory /NAS/dfs/dn/current/BP-1056675496-10.15.101.22-1402685851165.
   old LV = -55; old CTime = 0.
   new LV = -56; new CTime = 1418674697177
2014-12-15 15:28:57,228 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to Namenode/10.15.101.22:8022. Exiting.
java.io.IOException: EEXIST: File exists
 
 
To resolve this i manually have edited  the VERSION on dfs directories and changed the ctime field and version fields to read  and , replaced the namenode metadata backup that i have taken before upgrade.
 
Now when i bring up the cluster , i see namenode reports lot of blocks as missing, i tried to find out the blocks that were reported missing and i couldnt find it anywhere, please help to resolve this issue.
 
I cannot afford to wipe out everything on  DN/NN dfs directories  as the data in the cluster is heavily used.
 

Regards

Suresh
Cloudera Employee
Posts: 578
Registered: ‎01-20-2014

Re: Datanode issue after CDH5.2 upgrade

The upgrade to v5.2 is not the same as in earlier minor version upgrades.
There has been a version upgrade to HDFS metadata. Please undo the manual
change to the Datanodes. Then follow this upgrade guide

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_upgrade_to_cdh52....

Regards,
Gautam Gopalakrishnan
Contributor
Posts: 28
Registered: ‎03-04-2014

Re: Datanode issue after CDH5.2 upgrade

Gautam,

 

I have followed the same documentation and have even  mentioned that in my original note.

 

Regards

Suresh

 

 

Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: Datanode issue after CDH5.2 upgrade

Have you tried upgrading it again from cloudera console?  I had similar issues and after removing the cluster node and upgrading, adding it back resolved the issue without deleting any data.

 

It worked pretty smooth.

 

Let me know how it goes.

Announcements