question Re: Standby NameNode cant start in ambari cluster in Archives of Support Questions (Read Only)

Standby NameNode cant start in ambari cluster

mike_bronson7 — Mon, 23 Oct 2017 23:22:59 GMT

we are trying to start the "Standby NameNode (HDFS)" on master01 machine in ambari cluster version 2.6

and we cant start it

we get the following logs:

ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode. org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 13361263 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 13361262; expected file to go up to 13361312

what chould be the problem m and how to fix , so the service will start?

Re: Standby NameNode cant start in ambari cluster

jsensharma — Tue, 24 Oct 2017 00:05:58 GMT

@uri ben-ari

1. Was this Node working fine earlier?

2. Do you have the correct "etc/hosts" entry in upper case or mixed case.

3. Is this a kerberized cluster?

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 00:12:42 GMT

Was this Node working fine earlier - yes

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 00:14:35 GMT

Do you have the correct "etc/hosts" - yes

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 00:15:57 GMT

Is this a kerberized cluster? - what you mean we have 3 masters machine + 2 workers machines

Re: Standby NameNode cant start in ambari cluster

jsensharma — Tue, 24 Oct 2017 00:17:51 GMT

@uri ben-ari

If it is test cluster , then you may try the following (At your own risk)

1. If your Active NameNode is running fine then you can try to bring safe mode out with forceExit on Active NN.

2. Then take a backup of the directory "/hadoop/hdfs/namenode/current"

3. After taking the backup mentioned in previous step, please remove the directory contents "/hadoop/hdfs/namenode/current/*"

4. Perform the bootstrapStandby.

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 01:22:44 GMT

can you please show me how to bring safe mode out with forceExit on Active NN , bootstrapStandby.

Re: Standby NameNode cant start in ambari cluster

jsensharma — Tue, 24 Oct 2017 01:33:52 GMT

@uri ben-ari

Something like this you can try:

# su - hdfs 
# hdfs dfsadmin -safemode leave

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 01:50:45 GMT

my active node is on master02 so I need to do the steps on master02 machine? ( include backup )

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 13:08:22 GMT

hi Jay still waiting to your answer , do you mean to do hdfs dfsadmin -safemode leave on the namenode that is runing ( master02 ) ? ( standby namenode is on master01 ) , second - Then take a backup of the directory "/hadoop/hdfs/namenode/current" should be on the active name node ? , I ask this because it is more logic to do this on standby ( master01 machine )

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 15:27:06 GMT

we found this workaround - hadoop namenode -recover , is it other soultion for our problem ?

Re: Standby NameNode cant start in ambari cluster

mike_bronson7 — Tue, 24 Oct 2017 16:20:05 GMT

hi jay just to clarify your instructions ( and regarding that standby name node on master01 and active node on master02 can you approve this steps

su - hdfs

hdfs dfsadmin -safemode leave ( on master02 )

cp -rp /hadoop/hdfs/journal/hdfsha/current /hadoop/hdfs/journal/hdfsha/current.orig ( on master02 )

rm -f /hadoop/hdfs/journal/hdfsha/current/* ( on master02 )

hdfs namenode -bootstrapStandby ( on master01 )

Re: Standby NameNode cant start in ambari cluster

nishant_verma1 — Fri, 29 Dec 2017 15:25:52 GMT

I had created a new HA enabled cluster on EC2 instances. It came up without any issues. I then installed kerberos on two machines as master and slave, and then started the kerberos enable ambari wizard. At the last step of starting the services, Namenode start failed on both the nodes and gave same error as above. How this error came on a new setup and how to prevent it from coming up in my next environment setup?

Re: Standby NameNode cant start in ambari cluster

nishant_verma1 — Fri, 29 Dec 2017 15:26:35 GMT

Re: Standby NameNode cant start in ambari cluster

Koffi — Fri, 17 Dec 2021 16:00:59 GMT

Hi @mike_bronson7 ,

were you able to solve the issue with those steps?

Thank you