How to check if my Secondary Name Node is synced up with the cluster?

[#1]

Hi all,

This question is in continuation to: https://community.hortonworks.com/questions/138257/how-to-sync-a-new-secondary-namenode-to-the-clust...

Scenario: I have a 12 node cluster (machines 01-12) with 8 data nodes. 06, 07 are the NN and SNN respectively. 01 and 12 run the Hive related services. This cluster was upgraded to 12 node from 4 node where 01 was the namenode and 02-05 were the data nodes.

So I used the "Move Namenode" and "Move Snamenode" wizards on Ambari to move the nn and snn from 01 to 06 and 07 respectively. I verified all the services running on nn are running on snn as well. Now I want to check if my snn is working properly. So I shut down all the services on nn and tried to connect to Hive from one of the hosts (02) and it failed with the error:

Call From <machine_02/ip_address_of_02> to <machine_06:8020> failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

I don't know why it is trying to connect to nn (06) and not snn (07). Can someone point what I am missing here?

[#2]

Since you are talking about secondary NameNode, a Secondary NameNode will never act as a Name/meta data service provider even if you shutdown primary Namenode. You will have to switch Secondary Namenode to standby NameNode using HA.

You can read here - https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Secondary_N....

One way to check if your secondary NN has all latest fs image is by checking the size of CURRENT directory of NN and SNN.

[#3]

@Sree Kupp

I see you are attempting something that is taken care of by real HA setup with active and standby namenode, you can use below command to force failover.

$ hdfs haadmin -failover

Let me know whether that helps