Dynamically Add Hosts to a Cluster with Blueprints

[#1]

It's coming up more often now, the need to "add" host to an existing cluster without using the Ambari UI. I once thought the Blueprints were only useful to "initially" provision a cluster. But they're also quite helpful to extend your cluster as well.

Provision the Initial Cluster using Auto-Discovery

In the following example, we'll use a new feature in Ambari 2.1, called "Auto-Discovery" to provision the first node in our cluster.

Do NOT manually register the Host with Ambari Server yet. We're going to demonstrate "Auto-Discovery" to initialize new hosts.

Check that NO hosts have been registered with the cluster.

curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X GET http://<ambari-server:port>/api/v1/hosts

Blueprint has be registered with Ambari

  {
    "Blueprints": {
        "stack_name": "HDP",
        "stack_version": "2.3"
    },
    "configurations": [],
    "host_groups": [
        {
            "cardinality": "1",
            "components": [
                {
                    "name": "ZOOKEEPER_SERVER"
                },
                {
                    "name": "ZOOKEEPER_CLIENT"
                }
            ],
            "configurations": [],
            "name": "zookeeper_host_group"
        },
        {
            "cardinality": "1",
            "components": [
                {
                    "name": "KAFKA_BROKER"
                }
            ],
            "configurations": [],
            "name": "kafka_host_group"
		}
    ]
}

Save this above to a file and register the Blueprint with you cluster.

curl -i -H "X-Requested-By: ambari" -u admin:admin -X POST -d @myblueprint.json http://ambari-server:8080/api/vi/blueprints/basic1

Create your cluster

Now lets create our cluster and use the Auto-Discovery feature to provision the host(s). This is the 'template' file we use to provision the cluster.

{
  "blueprint" : "basic1",
  "host_groups" :[
    { 
      "name" : "zookeeper_host_group",   
      "host_count" : "1",
      "host_predicate" : "Hosts/cpu_count>0"     
    }
  ]
}

At this point the cluster has been created in Ambari, but you should NOT have any hosts. Ambari Server is sitting around waiting for a host to be registered with the cluster matching the "predicate" above. The "host_count" is also an important part of the criteria. The process will not start until the number of host in the "host_count" are available. For our demonstration here, we've set it to 1. In the real world, you would need 3 to create a valid ZooKeeper quorum.

Manually Register a Host (just one, for now)

Install an Ambari Agent on your host and configure it to talk to Ambari. The host should match the predicate above "cpu_count>0", which would be just about everything. But you get the point.

Start the Ambari-Agent and return back to the Ambari Server UI. In short order, you should see operations kicking off to provision the node and add the service. Now you have a cluster with 1 node and a running ZooKeeper Service.

Lets Expand our Cluster

With a second host, lets manually register the agent with Ambari. Check via the API that the host has been registered with the Ambari Server

curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X GET http://<ambari-server:port>/api/v1/hosts

Notice that the previously registered Blueprint "basic1", has a host group for Kafka Brokers. Let's add this newly registered host to our cluster as a Kakfa Broker.

Create a file to contain the message body and save it.

{
	"blueprint" : "basic1",
	"host_group" : "kafka_host_group"
}

Add and provision the host with the services specified in the Blueprint.

curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X POST -d @kafka_host.json http://<ambari-server:port>/api/v1/clusters/<yourcluster>/hosts/<new_host_id>;

The <new_host_id> is the name that it's been registered to Ambari by, should be the host FQDN. Double check the name returned by the hosts query above.

This will add the host to you new cluster and provision it with the services configured in the Blueprint.

Summary

In the past, Blueprints were used to initialized clusters. Now you can use them to "extend" your cluster. Avoid trying to reverse engineer the process of installing and configuring services through the REST API, use "Blueprints". You'll save yourself a lot of effort!!

References

Ambari Blueprints

[#2]

It’s an interesting fact that a huge data can be saved easily on a secure cloud space, and that is why we are able to operate e-commerce websites in a risk-free environment. However, the first step is to buy a relevant domain name, and then get a reliable web hosting platform that can handle heavy load with utter smoothness. I follow the suggestions from WebHostingGeeks.com whenever I create new websites for my clients.

[#3]

I have created a cluster using ambari UI.so Is it possible to scale it through Ambari Blueprint API?

When I am trying to scale (adding one node with services) it is showing the following message:

"message" : "Topology validation failed: org.apache.ambari.server.topology.InvalidTopologyException: Unable to retrieve cluster topology for cluster. This is most likely a result of trying to scale a cluster via the API which was created using the Ambari UI. At this time only clusters created via the API using a blueprint can be scaled with this API.

Please suggest me whether its possible or not to add nodes using API if my cluster was previously created by ambari UI?

[#4]

I have been trying your suggestion, added the host via the above API call and get the following:

18 May 2018 14:18:37,613  INFO [qtp-ambari-agent-326] TopologyManager:637 - TopologyManager.onHostRegistered: Entering
18 May 2018 14:18:37,613  INFO [qtp-ambari-agent-326] TopologyManager:698 - TopologyManager: Queueing available host node5
18 May 2018 14:19:37,959  WARN [alert-event-bus-1] AlertReceivedListener:497 - Unable to process alert ambari_agent_disk_usage for cluster idp and host node5 because the host is not a part of the cluster.
18 May 2018 14:20:06,779  INFO [ambari-client-thread-74] TopologyManager:485 - TopologyManager.scaleHosts: Entering
18 May 2018 14:20:06,779  INFO [ambari-client-thread-74] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1
18 May 2018 14:20:06,780  INFO [ambari-client-thread-74] HostRequest:205 - Skipping Start task creation since provision action = INSTALL_ONLY
18 May 2018 14:20:06,782  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for KAFKA_BROKER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,783  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for ZOOKEEPER_SERVER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,784  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for METRICS_MONITOR on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,784  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for NIFI_MASTER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,785  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for LOGSEARCH_LOGFEEDER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,786  INFO [ambari-client-thread-74] HostRequest:244 - Skipping create of START task for ZOOKEEPER_CLIENT on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:20:06,786  INFO [ambari-client-thread-74] HostRequest:99 - HostRequest: Created request for host: node5
18 May 2018 14:20:06,786  INFO [ambari-client-thread-74] LogicalRequest:437 - LogicalRequest.createHostRequests: all host requests size 1 , outstanding requests size = 0
18 May 2018 14:20:06,790  INFO [ambari-client-thread-74] TopologyManager:923 - TopologyManager.createLogicalRequest: created LogicalRequest with ID = 52 and completed persistence of this request.
18 May 2018 14:20:06,793  INFO [ambari-client-thread-74] TopologyManager:845 - TopologyManager.processRequest: Entering
18 May 2018 14:20:06,794  INFO [ambari-client-thread-74] TopologyManager:863 - TopologyManager.processRequest: host name = node5 is mapped to LogicalRequest ID = 52 and will be removed from the reserved hosts.
18 May 2018 14:20:06,794  INFO [ambari-client-thread-74] TopologyManager:876 - TopologyManager.processRequest: offering host name = node5 to LogicalRequest ID = 52
18 May 2018 14:20:06,794  INFO [ambari-client-thread-74] LogicalRequest:101 - LogicalRequest.offer: attempting to match a request to a request for a reserved host to hostname = node5
18 May 2018 14:20:06,794  INFO [ambari-client-thread-74] LogicalRequest:110 - LogicalRequest.offer: request mapping ACCEPTED for host = node5
18 May 2018 14:20:06,794  INFO [ambari-client-thread-74] LogicalRequest:113 - LogicalRequest.offer returning response, reservedHost list size = 0
18 May 2018 14:20:06,795  INFO [ambari-client-thread-74] TopologyManager:886 - TopologyManager.processRequest: host name = node5 was ACCEPTED by LogicalRequest ID = 52 , host has been removed from available hosts.
18 May 2018 14:20:06,795  INFO [ambari-client-thread-74] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1
18 May 2018 14:20:06,797  INFO [ambari-client-thread-74] TopologyManager:963 - TopologyManager.processAcceptedHostOffer: queue tasks for host = node5 which responded ACCEPTED
18 May 2018 14:20:06,797  INFO [ambari-client-thread-74] TopologyManager:988 - TopologyManager.processAcceptedHostOffer: queueing tasks for host = node5
18 May 2018 14:20:06,797  INFO [ambari-client-thread-74] TopologyManager:904 - TopologyManager.processRequest: not all required hosts have been matched, so adding LogicalRequest ID = 52 to outstanding requests

What can I do to move the request from PENDING to actually started?

Best