Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari blueprints and Namenode metadata

Solved Go to solution

Ambari blueprints and Namenode metadata

Contributor

A few questions on ambari blueprint and Rest API

Can someone please answer these questions?

1) After installing an HA cluster via blueprint, do we need to initialize the metadata on secondary namenode manually?

2) The same question in case we reboot the secondary namenode.

3) Can we monitor the status of Ambari agents through REST API?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ambari blueprints and Namenode metadata

If you install HDFS HA via blueprint, you dont have to initialize anything manually afterwards, its all done during the blueprint rollout.

You can monitor the status of Ambari Agents or hosts via

http://<ambari server>:8080/api/v1/hosts/<hostname>

This will return a lot of information about the host, e.g. disk info, running services (inkl. process id), last heartbeat of ambari agent, health status, etc.

{
  "href" : "http://example.com:8080/api/v1/hosts/horton01.example.com",
  "Hosts" : {
    "cluster_name" : "bigdata",
    "cpu_count" : 2,
    "desired_configs" : null,
    "disk_info" : [
      {
        "available" : "5922140",
        "device" : "/dev/vda1",
        "used" : "13670764",
        "percent" : "70%",
        "size" : "20641404",
        "type" : "ext3",
        "mountpoint" : "/"
      ...
    ],
    "host_health_report" : "",
    "host_name" : "horton01.cloud.hortonworks.com",
    "host_state" : "HEALTHY",
    "host_status" : "HEALTHY",
    "ip" : "172.24.68.17",
....
....

	"agentTimeStampAtReporting" : 1453826633797,
        "serverTimeStampAtReporting" : 1453826633829,
        "liveServices" : [
          ...
        ]
      },
      "umask" : 18,
      ....
    },
    "last_heartbeat_time" : 1453826643874,
    "last_registration_time" : 1452849291890,
    "os_arch" : "x86_64",
    "os_family" : "redhat6",
    "os_type" : "centos6",
    "ph_cpu_count" : 2,
    "public_host_name" : "horton01.example.com",
    "rack_info" : "/14",
    "recovery_report" : {
      "summary" : "DISABLED",
      "component_reports" : [ ]
    },
    "recovery_summary" : "DISABLED",
    "total_mem" : 7543576
  },
  "alerts_summary" : {
    "CRITICAL" : 0,
    "MAINTENANCE" : 0,
    "OK" : 18,
    "UNKNOWN" : 0,
    "WARNING" : 1
  }

View solution in original post

8 REPLIES 8
Highlighted

Re: Ambari blueprints and Namenode metadata

If you install HDFS HA via blueprint, you dont have to initialize anything manually afterwards, its all done during the blueprint rollout.

You can monitor the status of Ambari Agents or hosts via

http://<ambari server>:8080/api/v1/hosts/<hostname>

This will return a lot of information about the host, e.g. disk info, running services (inkl. process id), last heartbeat of ambari agent, health status, etc.

{
  "href" : "http://example.com:8080/api/v1/hosts/horton01.example.com",
  "Hosts" : {
    "cluster_name" : "bigdata",
    "cpu_count" : 2,
    "desired_configs" : null,
    "disk_info" : [
      {
        "available" : "5922140",
        "device" : "/dev/vda1",
        "used" : "13670764",
        "percent" : "70%",
        "size" : "20641404",
        "type" : "ext3",
        "mountpoint" : "/"
      ...
    ],
    "host_health_report" : "",
    "host_name" : "horton01.cloud.hortonworks.com",
    "host_state" : "HEALTHY",
    "host_status" : "HEALTHY",
    "ip" : "172.24.68.17",
....
....

	"agentTimeStampAtReporting" : 1453826633797,
        "serverTimeStampAtReporting" : 1453826633829,
        "liveServices" : [
          ...
        ]
      },
      "umask" : 18,
      ....
    },
    "last_heartbeat_time" : 1453826643874,
    "last_registration_time" : 1452849291890,
    "os_arch" : "x86_64",
    "os_family" : "redhat6",
    "os_type" : "centos6",
    "ph_cpu_count" : 2,
    "public_host_name" : "horton01.example.com",
    "rack_info" : "/14",
    "recovery_report" : {
      "summary" : "DISABLED",
      "component_reports" : [ ]
    },
    "recovery_summary" : "DISABLED",
    "total_mem" : 7543576
  },
  "alerts_summary" : {
    "CRITICAL" : 0,
    "MAINTENANCE" : 0,
    "OK" : 18,
    "UNKNOWN" : 0,
    "WARNING" : 1
  }

View solution in original post

Highlighted

Re: Ambari blueprints and Namenode metadata

Contributor

@Jonas Straub Thanks for your quick reply but could you also answer the question # 2

The same question in case we reboot the secondary namenode?

Highlighted

Re: Ambari blueprints and Namenode metadata

Just to avoid confusion, there is no Secondary Namenode, there will only be a Namenode-1 and Namenode-2, one of these two namenodes is always the active one, the other one will be a standby NN.

During the Blueprint rollout Ambari will execute several steps (see here) to initialize the Journalnodes, format ZK Znode and distribute all the metadata (fsimage,etc.). These steps are only executed once. If you restart an active Namenode, it will transition to a standby state first and make the other NN the active one, once thats done it restarts, so the metadata is not reinitialized again.

Highlighted

Re: Ambari blueprints and Namenode metadata

Mentor

@rbalam has this been resolved? Can you provide a solution or accept the best answer?

Highlighted

Re: Ambari blueprints and Namenode metadata

Contributor

"see here" link is unavailable now. Could you please have an updated link? @Jonas Straub

I am installing HDP3.0 HA using ambari 2.7 blueprint. The namenodes failed to start due to error "NameNode is not formatted." This did not happen to HDP 2.6 using almost the same blueprint. Look like ambari rollout failed to distribute the metadata. Is there way to get the root cause? The log did not help much. Thanks.

Highlighted

Re: Ambari blueprints and Namenode metadata

Mentor

@Lian Jiang

Can you share your blueprint after stripping iit of site specific info?

Highlighted

Re: Ambari blueprints and Namenode metadata

Contributor

Geoffrey,

blueprint for HDP3.0

Thanks.

Highlighted

Re: Ambari blueprints and Namenode metadata

Contributor

By understanding ambari source code, the problem is solved. I need to specify below in blueprint:

	 "cluster-env": {
	  "properties": {
	  "dfs_ha_initial_namenode_active": "%HOSTGROUP::master_host_group%",
	  "dfs_ha_initial_namenode_standby": "%HOSTGROUP::master2_host_group%"
	  }
	 }
	 

Don't have an account?
Coming from Hortonworks? Activate your account here