question Re: how to force name node to be active in Support Questions

how to force name node to be active

mike_bronson7 — Sun, 18 Aug 2019 03:09:06 GMT

in our ambari cluster both name node are like standby

in order to force one of them to be active we do

 hdfs haadmin -transitionToActive --forceactive master01 

Illegal argument: Unable to determine service address for namenode 'master01'

but we get - Unable to determine service address

what this is indicate ? and how to fix this issue ?

Re: how to force name node to be active

Shelton — Mon, 04 Dec 2017 19:16:39 GMT

@Michael Bronson

The correct syntax should be

haadmin -failover -forceactive namenode1(active) namenode1(standby)

Note the active and standby inputs

Hope that helps

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 19:28:43 GMT

we get that:

hdfs haadmin -failover -forceactive master01 master03

Illegal argument: Unable to determine service address for namenode 'master01'

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 19:31:27 GMT

# host master01 
master01.sys4.com has address 103.114.28.13 
# host master03
master03.sys4.com has address 103.114.28.12

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 19:39:32 GMT

why we get - Unable to determine service address for namenode ?

Re: how to force name node to be active

Shelton — Mon, 04 Dec 2017 20:02:39 GMT

@Michael Bronson

From your screenshot, both namenodes are down hence the failure of the failover commands. Since you enabled NameNode HA using Ambari and the ZooKeeper service instances and ZooKeeper FailoverControllers to be up and running.

Just restart the name nodes but its bizarre that none is marked (Active and Standby). Depending on the cluster use DEV or Prod please take the appropriate steps to restart the namenode because your cluster is now unusable anyway.

Using Ambari use the HDFS restart all command under Service actions ,

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 20:15:41 GMT

@Geoffrey , I already restart the HDFS but without good results after restart the status is as the current status , so I just thinking what are the next steps to resolve this problem ?

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 20:22:07 GMT

@Geoffrey , according to my bad status , what else we can do , ( yesterday we restart all machines in the cluster , and start the services from Zk and HDFS and so on ) , so I am really stuck here

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 20:46:07 GMT

we see from the logs also this

017-12-04 12:37:33,544 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [100.14.28.153:8485, 100.14.28.152:8485, 100.14.27.162:8485], stream=null))
java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond.

Re: how to force name node to be active

Shelton — Mon, 04 Dec 2017 21:25:19 GMT

@Michael Bronson

If this is a production environment I would advise you to contact hortonworks support.

How many nodes in your cluster?

How many Journalnodes you have in cluster ? Make sure you have odd number.

Could you also confirm whether at any point after enabling the HA the Active and Standby namenodes ever functioned?

Your log messages indicates that there was a timeout condition when the NameNode attempted to call the JournalNodes. The NameNode must successfully call a quorum of JournalNodes: at least 2 out of 3. This means that the call timed out to at least 2 out of 3 of them.

This is a fatal condition for the NameNode, so by design, it aborts.

There are multiple potential reasons for this timeout condition. Reviewing logs from the NameNodes and JournalNodes would likely reveal more details.

If its a none critical cluster ,you can follow the below steps

Stop the Hdfs service if its running.

Start only the journal nodes (as they will need to be made aware of the formatting)

On the first namenode (as user hdfs)

# su - hdfs

Format the namenode

$ hadoop namenode -format

Initialize the Edits (for the journal nodes)

$ hdfs namenode -initializeSharedEdits -force

Format Zookeeper (to force zookeeper to reinitialise)

$ hdfs zkfc -formatZK -force

Using Ambari restart that first namenode

On the second namenode Sync (force synch with first namenode)

$ hdfs namenode -bootstrapStandby -force

On every datanode clear the data directory

Restart the HDFS service

Hope that helps

Re: how to force name node to be active

mike_bronson7 — Mon, 04 Dec 2017 22:25:51 GMT

on the namenode -format we get -

17/12/04 14:23:34 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Timed out waiting for response from loggers

Re: how to force name node to be active

Shelton — Tue, 05 Dec 2017 06:00:37 GMT

@Michael Bronson

Can you attach your namenode log How's your /etc/hosts entry?

103.114.28.13  master01.sys4.com 
103.114.28.12 master03.sys4.com

or IP/hostname /Alias

103.114.28.13  master01.sys4.com   master01
103.114.28.12 master03.sys4.com    master03

What is the output of

$ zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls 
/hadoop-ha

If this cluster is not critical then you might have to have to go through these steps

Re: how to force name node to be active

mike_bronson7 — Sun, 18 Aug 2019 03:08:58 GMT

I do it on the first machine - master01 ( this is standby machine )

localhost:2181(CONNECTED) 2] ls /hadoop-ha

[hdfsha]

Re: how to force name node to be active

mike_bronson7 — Tue, 05 Dec 2017 06:14:44 GMT

my feeling is that no mater which namenode we start , every namenode became to standby and this is the big problem

Re: how to force name node to be active

Shelton — Tue, 05 Dec 2017 06:22:40 GMT

@Michael Bronson

localhost:2181(CONNECTED) 2] ls /hadoop-ha
[hdfsha]

localhost:2181(CONNECTED) 2] get /hadoop-ha/hdfsha/ActiveStandbyElectorLock

What output do you get ?

Re: how to force name node to be active

mike_bronson7 — Tue, 05 Dec 2017 06:23:28 GMT

regarding to the host file , we not use it , we have DNS server , and all hosts are resolved . we already check that , and all ip's point to the right hostnames

Re: how to force name node to be active

mike_bronson7 — Tue, 05 Dec 2017 06:25:51 GMT

[zk: localhost:2181(CONNECTED) 6] ls /hadoop-ha/hdfsha/ActiveStandbyElectorLock
Node does not exist: /hadoop-ha/hdfsha/ActiveStandbyElectorLock
[zk
: localhost:2181(CONNECTED) 7] get /hadoop-ha/hdfsha/ActiveStandbyElectorLock 
Node does not exist: /hadoop-ha/hdfsha/ActiveStandbyElectorLock
[zk: 
localhost:2181(CONNECTED) 8] ls /hadoop-ha/hdfsha 
[]
[zk: localhost:2181(CONNECTED) 9]

Re: how to force name node to be active

Shelton — Tue, 05 Dec 2017 06:34:23 GMT

@Michael Bronson

Can you delete the entry in zookeeper and restart

[zk: localhost:2181(CONNECTED) 1] rmr /hadoop-ha

Validate that there is no hadoop-ha entry,

[zk: localhost:2181(CONNECTED) 2] ls /

Then restart the all components HDFS service. This will create a new ZNode with correct lock(of Failover controller).

Also see https://community.hortonworks.com/questions/12942/how-to-clean-up-files-in-zookeeper-directory.html#

Re: how to force name node to be active

mike_bronson7 — Tue, 05 Dec 2017 06:38:42 GMT

we removed the /hadoop-ha from master01 machine , ans restart HDFS , I will update soon , from my expiriance its takes time around 20min

Re: how to force name node to be active

mike_bronson7 — Sun, 18 Aug 2019 03:08:50 GMT

this is the status after full HDFS restart as you see we get both standby -:(