question Re: blockpoolID in VERSION Files ... similar or different in Archives of Support Questions (Read Only)

blockpoolID in VERSION Files ... similar or different

luc_tiber — Thu, 28 Jan 2016 00:10:58 GMT

I am running a small mutlinode env on Hadoop 2.7.1/Ubuntu (1master+1secNN+3slaves)

While troubleshooting Datanodes and NameNodes, I noticed a discrepancy in the VERSION files.

On the 2NN and the 3 slaves, the blockpoolID parameter is:

blockpoolID=BP-1913131753-XXX.XX.XX.169-1452777222838 BUT on the master the blockpoolID is BP-90521626-XXX.XX.XX.169-1453822984214.

Shouldn´t all be the same? if yes should I manually override the 2NN and 3 slaves VERSION blockpoolID with master one?

thanks

Re: blockpoolID in VERSION Files ... similar or different

ArpitAgarwal — Thu, 28 Jan 2016 03:54:24 GMT

Hi @luc tiber, the BlockpoolID should be the same across the DataNodes and the NameNodes (the configuration is a little different for federated clusters so let's ignore federation for now).

The fact that the BPID is different indicates that the NameNode was either re-formatted or belongs to a different cluster, so just changing the BPID will not help as the NameNode metadata may have been lost.

If this a dev/test cluster and you can afford to lose the data, I recommend redeploying the cluster from scratch as the quickest way to move forward. Else recovery will be more complicated.

Re: blockpoolID in VERSION Files ... similar or different

luc_tiber — Thu, 28 Jan 2016 14:11:21 GMT

Hi Arpit

Thanks for your recommendation . Since I am using AWS, what do you think about creating an AMI from my master and then creating the slaves based on this AMI ?

Re: blockpoolID in VERSION Files ... similar or different

ArpitAgarwal — Fri, 29 Jan 2016 05:26:00 GMT

Hi @luc tiber, the HDFS metadata layout on NameNodes and DataNodes is quite different. If you are using HDP I recommend doing an Ambari-based install.

Re: blockpoolID in VERSION Files ... similar or different

luc_tiber — Fri, 29 Jan 2016 14:52:43 GMT

Thanks Aprit for your feedbac

Thanks Aprit for your recommendation