blockpoolID in VERSION Files ... similar or different

[#1]

I am running a small mutlinode env on Hadoop 2.7.1/Ubuntu (1master+1secNN+3slaves)

While troubleshooting Datanodes and NameNodes, I noticed a discrepancy in the VERSION files.

On the 2NN and the 3 slaves, the blockpoolID parameter is:

blockpoolID=BP-1913131753-XXX.XX.XX.169-1452777222838 BUT on the master the blockpoolID is BP-90521626-XXX.XX.XX.169-1453822984214. 

Shouldn´t all be the same? if yes should I manually override the 2NN and 3 slaves VERSION blockpoolID with master one?

thanks

[#2 · Accepted solution]

Hi @luc tiber, the BlockpoolID should be the same across the DataNodes and the NameNodes (the configuration is a little different for federated clusters so let's ignore federation for now).

The fact that the BPID is different indicates that the NameNode was either re-formatted or belongs to a different cluster, so just changing the BPID will not help as the NameNode metadata may have been lost.

If this a dev/test cluster and you can afford to lose the data, I recommend redeploying the cluster from scratch as the quickest way to move forward. Else recovery will be more complicated.

[#3]

Hi Arpit

Thanks for your recommendation . Since I am using AWS, what do you think about creating an AMI from my master and then creating the slaves based on this AMI ?

[#4]

Hi @luc tiber, the HDFS metadata layout on NameNodes and DataNodes is quite different. If you are using HDP I recommend doing an Ambari-based install.

[#5]

Thanks Aprit for your feedbac

,

Thanks Aprit for your recommendation