How to write Cluster Startup Shutdown order and Scripts with Ambari

[#1]

I am running HDP on a Virtualized environment and for business reasons needs to be able to shut down/restart the cluster if necessary.

Need to figure out how to write a shutdown script. Which order should the services be started or shut down?

[#2 · Accepted solution]

Here is a start.

Assuming HDP 2.2.x an order of services to shutdown (except name node ) is found at: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.8/bk_HDP_Reference_Guide/content/stopping_hdp_...

The order to startup: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.8/bk_HDP_Reference_Guide/content/starting_hdp_...For HDP 2.3 -> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_Reference_Guide/content/ch_controllin...

Here is a sample control script you can use to start stop services: https://gist.github.com/randerzander/5b7b0e075e59f87d3c84

Also read the Ambari API docs as suggested below on the Ambari APIs: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=41812517 --------------------------

export $PASSWORD = *****

export $CLUSTER_NAME = {Your Cluster Name}

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start HDFS via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/HDFS

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start YARN via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/YARN

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start MAPREDUCE2 via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/MAPREDUCE2

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start HIVE via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/HIVE

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start TEZ via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/TEZ

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start HCATALOG via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/HCATALOG

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start WEBHCAT via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/WEBHCAT

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start ZOOKEEPER via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/ZOOKEEPER

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start NAGIOS via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/NAGIOS

curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'-X PUT -d '{"RequestInfo": {"context" :"Start GANGLIA via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}'http://`hostname -f`:8080/api/v1/clusters/$CLUSTER_NAME/services/GANGLIA

[#3]

Would not recommend creating these scripts manually because it is specific to Amabri version and you would need to change this script every release as new services are added as start order may change. Ambari knows the right order (check the role_command_order.json files under /var/lib/ambari-server). I would recommend to just use Ambaris APIs to start all and stop all so you don't have to worry about it every release. @yusaku@hortonworks.com what do you think?

[#4]

Might be easier to invoke the Ambari APIs to stop/start all services. To do this at startup invoke these from your /etc/rc.d/rc.local:

USER=admin
PASSWORD=admin
AMBARI_HOST=localhost

#detect name of cluster
output=`curl -s -u $USER:$PASSWORD -i -H 'X-Requested-By: ambari'  http://$AMBARI_HOST:8080/api/v1/clusters`
CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`

#stop all services
curl -u $USER:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d  '{"RequestInfo":{"context":"_PARSE_.STOP.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"Sandbox"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services

#start all services
curl -u $USER:$PASSWORD -i -H 'X-Requested-By: ambari' -X PUT -d  '{"RequestInfo":{"context":"_PARSE_.START.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"Sandbox"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services

Instead of starting all services, you can also start services individually. See below example that starts only Kafka, Hbase, Storm

USER=admin
PASSWORD=admin
AMBARI_HOST=localhost

#detect name of cluster
output=`curl -s -u $USER:$PASSWORD -i -H 'X-Requested-By: ambari'  http://$AMBARI_HOST:8080/api/v1/clusters`
CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`

for SERVICE in KAFKA HBASE STORM
do
	echo "starting $SERVICE"
	curl -u $USER:$PASSWORD -i -H "X-Requested-By: ambari" -X PUT -d "{\"RequestInfo\": {\"context\" :\"Start $SERVICE via REST\"}, \"Body\": {\"ServiceInfo\": {\"state\": \"STARTED\"}}}" http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER/services/$SERVICE
done	

[#5]

Be cautious. The assumption with all techniques using the Ambari REST API to auto start the cluster is the notion that the ambari-server process is running and also reachable via the network. I have seen cases in AWS where the network connection is not yet available to the Ambari server host, resulting in the REST calls failing.