Name Node services fail over

[#1]

I need to know if there was any failover happened on name node services. Where can I get this information.

Is there any REST API ?

[#2]

@Kumar Veerappan

The Following kind of Ambari API call can show which NameNode is Active and which one is StandBy.

Syntax:

curl -iv -u admin:admin -H "X-Requested-By: ambari" -X GET  "http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/components/NAMENODE?ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/service_name,host_components/metrics/dfs/FSNamesystem/HAState"

Example:

curl -iv -u admin:admin -H "X-Requested-By: ambari" -X GET  "http://amb25101.example.com:8080/api/v1/clusters/plain_ambari/components/NAMENODE?ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/service_name,host_components/metrics/dfs/FSNamesystem/HAState"

.

Example Output of the above call might look like:

{
  "href" : "http://amb25101.example.com:8080/api/v1/clusters/plain_ambari/components/NAMENODE?ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/service_name,host_components/metrics/dfs/FSNamesystem/HAState",
  "ServiceComponentInfo" : {
    "category" : "MASTER",
    "cluster_name" : "plain_ambari",
    "component_name" : "NAMENODE",
    "service_name" : "HDFS"
  },
  "host_components" : [
    {
      "href" : "http://amb25101.example.com:8080/api/v1/clusters/plain_ambari/hosts/amb25101.example.com/host_components/NAMENODE",
      "HostRoles" : {
        "cluster_name" : "plain_ambari",
        "component_name" : "NAMENODE",
        "host_name" : "amb25101.example.com"
      },
      "metrics" : {
        "dfs" : {
          "FSNamesystem" : {
            "HAState" : "active"
          }
        }
      }
    },
    {
      "href" : "http://amb25101.example.com:8080/api/v1/clusters/plain_ambari/hosts/amb25102.example.com/host_components/NAMENODE",
      "HostRoles" : {
        "cluster_name" : "plain_ambari",
        "component_name" : "NAMENODE",
        "host_name" : "amb25102.example.com"
      },
      "metrics" : {
        "dfs" : {
          "FSNamesystem" : {
            "HAState" : "standby"
          }
        }
      }
    }
  ]
* Connection #0 to host amb25101.example.com left intact
}

.

[#3]

@Kumar Veerappan

You will have to create an watcher/alert scripts that identifies which NN is active and alert/email if NN flips.

Namenode service provides JMX which has information on which NameNode is active. Your watcher script can query this data to identify if NN failovers.

name" : "Hadoop:service=NameNode,name=NameNodeStatus",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
    "State" : "active",
    "NNRole" : "NameNode",
    "HostAndPort" : "host1:8020

[#4 · Accepted solution]

Fount out that this is available in zkfc logs which will be present in the name node.

For some reason these logs are not present in my name node boxes, so probably I need to restart the ZKFC services.

Thanks

Kumar