Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Standby NameNode cant start in ambari cluster

Solved Go to solution

Standby NameNode cant start in ambari cluster

we are trying to start the "Standby NameNode (HDFS)" on master01 machine in ambari cluster version 2.6

and we cant start it

we get the following logs:

ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode. org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 13361263 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:693) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:703) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1701) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1769) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 13361262; expected file to go up to 13361312

what chould be the problem m and how to fix , so the service will start?

Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Standby NameNode cant start in ambari cluster

Super Mentor

@uri ben-ari

1. Was this Node working fine earlier?

2. Do you have the correct "etc/hosts" entry in upper case or mixed case.

3. Is this a kerberized cluster?


13 REPLIES 13

Re: Standby NameNode cant start in ambari cluster

Super Mentor

@uri ben-ari

1. Was this Node working fine earlier?

2. Do you have the correct "etc/hosts" entry in upper case or mixed case.

3. Is this a kerberized cluster?


Re: Standby NameNode cant start in ambari cluster

Was this Node working fine earlier - yes

Michael-Bronson

Re: Standby NameNode cant start in ambari cluster

Do you have the correct "etc/hosts" - yes

Michael-Bronson

Re: Standby NameNode cant start in ambari cluster

Is this a kerberized cluster? - what you mean we have 3 masters machine + 2 workers machines

Michael-Bronson

Re: Standby NameNode cant start in ambari cluster

Super Mentor

@uri ben-ari

If it is test cluster , then you may try the following (At your own risk)

1. If your Active NameNode is running fine then you can try to bring safe mode out with forceExit on Active NN.

2. Then take a backup of the directory "/hadoop/hdfs/namenode/current"

3. After taking the backup mentioned in previous step, please remove the directory contents "/hadoop/hdfs/namenode/current/*"

4. Perform the bootstrapStandby.

.

Re: Standby NameNode cant start in ambari cluster

can you please show me how to bring safe mode out with forceExit on Active NN , bootstrapStandby.

Michael-Bronson

Re: Standby NameNode cant start in ambari cluster

Super Mentor

@uri ben-ari

Something like this you can try:

# su - hdfs 
# hdfs dfsadmin -safemode leave

.

Re: Standby NameNode cant start in ambari cluster

my active node is on master02 so I need to do the steps on master02 machine? ( include backup )

Michael-Bronson

Re: Standby NameNode cant start in ambari cluster

hi Jay still waiting to your answer , do you mean to do hdfs dfsadmin -safemode leave on the namenode that is runing ( master02 ) ? ( standby namenode is on master01 ) , second - Then take a backup of the directory "/hadoop/hdfs/namenode/current" should be on the active name node ? , I ask this because it is more logic to do this on standby ( master01 machine )

Michael-Bronson
Don't have an account?
Coming from Hortonworks? Activate your account here