Determine Ambari Host and Cluster Name from command line for configs.sh

[#1]

Hi,

I'm writing a script that will set up the necessary classpath on a Spark cluster managed by Ambari.

Originally, I had a script that would just edit /etc/spark/conf/spark-defaults.conf directly. However, as discussed here, that was bad solution since Ambari regularly overwrites that file with whatever configuration is registered with Ambari.

Therefore, I am now using the configs.sh script as described here to make these adjustments.

I am looking for a way to determine the AMBARI_HOST and CLUSTER_NAME arguments via the command line. Here is what I have so far:

#!/usr/bin/env bash

ambari_host=  # TODO: some command to find this out
cluster_name= # TODO: some command to find this out
config_types=("spark.executor.extraClassPath" "spark.driver.extraClassPath")
phoenix_jar="$(ls /usr/hdp/current/phoenix-client/phoenix-*-client-spark.jar)"
sqljdbc_jar="$(ls /usr/share/java/sqljdbc*.jar)"
jars="$phoenix_jar:$sqljdbc_jar"

for config_type in "${config_types[@]}"; do
    /var/lib/ambari-server/resources/scripts/configs.sh \
	set "$ambari_host" "$cluster_name" spark-defaults "$config_type" "$jars"
done

I can elaborate on anything that's unclear. Thanks in advance!

Edit: thank you everyone for the answers! I wish I could accept all of them.

For posterity, here's what I ended up with:

#!/usr/bin/env bash

ambari_user="$1"
ambari_password="$2"
ambari_port=8080

ambari_host="$(/opt/hostname_scripts/hostname.sh)"
cluster_name="$(curl -u ${ambari_user}:${ambari_password} -i -H 'X-Requested-By: ambari'  http://$ambari_host:$ambari_port/api/v1/clusters | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p')"

config_types=("spark.executor.extraClassPath" "spark.driver.extraClassPath")

phoenix_jar="$(ls /usr/hdp/current/phoenix-client/phoenix-*-client-spark.jar)"
sqljdbc_jar="$(ls /usr/share/java/sqljdbc*.jar)"
jars="$phoenix_jar:$sqljdbc_jar"

for config_type in "${config_types[@]}"; do
    /var/lib/ambari-server/resources/scripts/configs.sh -u "$ambari_user" -p "$ambari_password" -port "$ambari_port" set "$ambari_host" "$cluster_name" spark-defaults "$config_type" "$jars" > /dev/null
done 

This was also a valuable resource.

[1] https://community.hortonworks.com/questions/43587/spark-defulatsconf-constantly-being-overwritten.ht...

[2] https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations#Modifyconfigurations-Editco...

[3] http://lecluster.delaurent.com/one-shot-backup-all-config-files-with-ambari-api/

[#2]

@Zach Kirsch

1) I don't know what you mean by AMBARI_HOST? Is it where your ambari server is installed? From any of the agent hosts, you can find the server host from : /etc/ambari-agent/conf/ambari-agent.ini.

Look for the entry [server] hostname=$SERVER_HOST

2) Are you ok using API?

You can get the cluster name by running :

curl --user admin:admin http://$AMBARI_HOST:8080/api/v1/clusters/

This will return you a response of the type :

{ "href" : "http://$AMBARI_HOST:8080/api/v1/clusters/", "items" : [ { "href" : "http://$AMBARI_HOST:8080/api/v1/clusters/clustername", "Clusters" : { "cluster_name" : "clustername", "version" : "version" } } ]}

You may then extract the cluster name.

Hope this helps!

[#3]

@Zach Kirsch

1) To find out cluster_name, you can either use the api curl --user admin:admin http://$AMBARI_SERVER:8080/api/v1/clusters/ as mentioned by Shreya,

Or you can query the database for table 'clusters' as below :

select cluster_name from clusters;

2) If you are looking to find out ambari server host, then as @sbhat mentioned, you can grep ambari-agent.ini. Below command would do :

grep "hostname" /etc/ambari-agent/conf/ambari-agent.ini

But if you are looking for the host on which particular spark component is installed, then please use this api :

curl --user admin:admin http://$AMBARI_SERVER:8080/api/v1/clusters/<cluster_name>/services/SPARK/<component_name>

From the response, parse the json for the field 'host_components'

[#4 · Accepted solution]

@Zach Kirsch here is how I usually detect the cluster name using parsing the json output via sed

export SERVICE=ZEPPELIN
export PASSWORD=admin
export AMBARI_HOST=localhost

#detect name of cluster
output=`curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'  http://$AMBARI_HOST:8080/api/v1/clusters`

CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`

echo $CLUSTER

[#5]

@Zach Kirsch

You can get Ambari server hostname from ambari-agent.ini file.

example:

[root@kknode3 ~]# cat /etc/ambari-agent/conf/ambari-agent.ini |grep -i host
hostname = kknode1.openstacklocal