Dynamically Add Hosts to a Cluster with Blueprints
It's coming up more often now, the need to "add" host to an existing cluster without using the Ambari UI. I once thought the Blueprints were only useful to "initially" provision a cluster. But they're also quite helpful to extend your cluster as well.
Provision the Initial Cluster using Auto-Discovery
In the following example, we'll use a new feature in Ambari 2.1, called "Auto-Discovery" to provision the first node in our cluster.
Do NOT manually register the Host with Ambari Server yet. We're going to demonstrate "Auto-Discovery" to initialize new hosts.
Check that NO hosts have been registered with the cluster.
curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X GET http://<ambari-server:port>/api/v1/hosts
{
"Blueprints": {
"stack_name": "HDP",
"stack_version": "2.3"
},
"configurations": [],
"host_groups": [
{
"cardinality": "1",
"components": [
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"configurations": [],
"name": "zookeeper_host_group"
},
{
"cardinality": "1",
"components": [
{
"name": "KAFKA_BROKER"
}
],
"configurations": [],
"name": "kafka_host_group"
}
]
}
Save this above to a file and register the Blueprint with you cluster.
curl -i -H "X-Requested-By: ambari" -u admin:admin -X POST -d @myblueprint.json http://ambari-server:8080/api/vi/blueprints/basic1
Create your cluster
Now lets create our cluster and use the Auto-Discovery feature to provision the host(s). This is the 'template' file we use to provision the cluster.
{
"blueprint" : "basic1",
"host_groups" :[
{
"name" : "zookeeper_host_group",
"host_count" : "1",
"host_predicate" : "Hosts/cpu_count>0"
}
]
}
At this point the cluster has been created in Ambari, but you should NOT have any hosts. Ambari Server is sitting around waiting for a host to be registered with the cluster matching the "predicate" above. The "host_count" is also an important part of the criteria. The process will not start until the number of host in the "host_count" are available. For our demonstration here, we've set it to 1. In the real world, you would need 3 to create a valid ZooKeeper quorum.
Manually Register a Host (just one, for now)
Install an Ambari Agent on your host and configure it to talk to Ambari. The host should match the predicate above "cpu_count>0", which would be just about everything. But you get the point.
Start the Ambari-Agent and return back to the Ambari Server UI. In short order, you should see operations kicking off to provision the node and add the service. Now you have a cluster with 1 node and a running ZooKeeper Service.
Lets Expand our Cluster
With a second host, lets manually register the agent with Ambari. Check via the API that the host has been registered with the Ambari Server
curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X GET http://<ambari-server:port>/api/v1/hosts
Notice that the previously registered Blueprint "basic1", has a host group for Kafka Brokers. Let's add this newly registered host to our cluster as a Kakfa Broker.
Create a file to contain the message body and save it.
{
"blueprint" : "basic1",
"host_group" : "kafka_host_group"
}
Add and provision the host with the services specified in the Blueprint.
curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X POST -d @kafka_host.json http://<ambari-server:port>/api/v1/clusters/<yourcluster>/hosts/<new_host_id>;
The <new_host_id> is the name that it's been registered to Ambari by, should be the host FQDN. Double check the name returned by the hosts query above.
This will add the host to you new cluster and provision it with the services configured in the Blueprint.
Summary
In the past, Blueprints were used to initialized clusters. Now you can use them to "extend" your cluster. Avoid trying to reverse engineer the process of installing and configuring services through the REST API, use "Blueprints". You'll save yourself a lot of effort!!
