Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Determine Ambari Host and Cluster Name from command line for configs.sh

avatar
Contributor

Hi,

I'm writing a script that will set up the necessary classpath on a Spark cluster managed by Ambari.

Originally, I had a script that would just edit /etc/spark/conf/spark-defaults.conf directly. However, as discussed here, that was bad solution since Ambari regularly overwrites that file with whatever configuration is registered with Ambari.

Therefore, I am now using the configs.sh script as described here to make these adjustments.

I am looking for a way to determine the AMBARI_HOST and CLUSTER_NAME arguments via the command line. Here is what I have so far:

#!/usr/bin/env bash

ambari_host=  # TODO: some command to find this out
cluster_name= # TODO: some command to find this out
config_types=("spark.executor.extraClassPath" "spark.driver.extraClassPath")
phoenix_jar="$(ls /usr/hdp/current/phoenix-client/phoenix-*-client-spark.jar)"
sqljdbc_jar="$(ls /usr/share/java/sqljdbc*.jar)"
jars="$phoenix_jar:$sqljdbc_jar"

for config_type in "${config_types[@]}"; do
    /var/lib/ambari-server/resources/scripts/configs.sh \
	set "$ambari_host" "$cluster_name" spark-defaults "$config_type" "$jars"
done

I can elaborate on anything that's unclear. Thanks in advance!

Edit: thank you everyone for the answers! I wish I could accept all of them.

For posterity, here's what I ended up with:

#!/usr/bin/env bash

ambari_user="$1"
ambari_password="$2"
ambari_port=8080

ambari_host="$(/opt/hostname_scripts/hostname.sh)"
cluster_name="$(curl -u ${ambari_user}:${ambari_password} -i -H 'X-Requested-By: ambari'  http://$ambari_host:$ambari_port/api/v1/clusters | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p')"

config_types=("spark.executor.extraClassPath" "spark.driver.extraClassPath")

phoenix_jar="$(ls /usr/hdp/current/phoenix-client/phoenix-*-client-spark.jar)"
sqljdbc_jar="$(ls /usr/share/java/sqljdbc*.jar)"
jars="$phoenix_jar:$sqljdbc_jar"

for config_type in "${config_types[@]}"; do
    /var/lib/ambari-server/resources/scripts/configs.sh -u "$ambari_user" -p "$ambari_password" -port "$ambari_port" set "$ambari_host" "$cluster_name" spark-defaults "$config_type" "$jars" > /dev/null
done 

This was also a valuable resource.

[1] https://community.hortonworks.com/questions/43587/spark-defulatsconf-constantly-being-overwritten.ht...

[2] https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations#Modifyconfigurations-Editco...

[3] http://lecluster.delaurent.com/one-shot-backup-all-config-files-with-ambari-api/

1 ACCEPTED SOLUTION

avatar

@Zach Kirsch here is how I usually detect the cluster name using parsing the json output via sed

export SERVICE=ZEPPELIN
export PASSWORD=admin
export AMBARI_HOST=localhost

#detect name of cluster
output=`curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'  http://$AMBARI_HOST:8080/api/v1/clusters`

CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`

echo $CLUSTER

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

@Zach Kirsch

1) I don't know what you mean by AMBARI_HOST? Is it where your ambari server is installed? From any of the agent hosts, you can find the server host from : /etc/ambari-agent/conf/ambari-agent.ini.

Look for the entry [server] hostname=$SERVER_HOST

2) Are you ok using API?

You can get the cluster name by running :

curl --user admin:admin http://$AMBARI_HOST:8080/api/v1/clusters/

This will return you a response of the type :

{ "href" : "http://$AMBARI_HOST:8080/api/v1/clusters/", "items" : [ { "href" : "http://$AMBARI_HOST:8080/api/v1/clusters/clustername", "Clusters" : { "cluster_name" : "clustername", "version" : "version" } } ]}

You may then extract the cluster name.

Hope this helps!

avatar
Super Collaborator
@Zach Kirsch

1) To find out cluster_name, you can either use the api curl --user admin:admin http://$AMBARI_SERVER:8080/api/v1/clusters/ as mentioned by Shreya,

Or you can query the database for table 'clusters' as below :

select cluster_name from clusters;

2) If you are looking to find out ambari server host, then as @sbhat mentioned, you can grep ambari-agent.ini. Below command would do :

grep "hostname" /etc/ambari-agent/conf/ambari-agent.ini

But if you are looking for the host on which particular spark component is installed, then please use this api :

curl --user admin:admin http://$AMBARI_SERVER:8080/api/v1/clusters/<cluster_name>/services/SPARK/<component_name>

From the response, parse the json for the field 'host_components'

avatar

@Zach Kirsch here is how I usually detect the cluster name using parsing the json output via sed

export SERVICE=ZEPPELIN
export PASSWORD=admin
export AMBARI_HOST=localhost

#detect name of cluster
output=`curl -u admin:$PASSWORD -i -H 'X-Requested-By: ambari'  http://$AMBARI_HOST:8080/api/v1/clusters`

CLUSTER=`echo $output | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p'`

echo $CLUSTER

avatar
Master Guru
@Zach Kirsch

You can get Ambari server hostname from ambari-agent.ini file.

example:

[root@kknode3 ~]# cat /etc/ambari-agent/conf/ambari-agent.ini |grep -i host
hostname = kknode1.openstacklocal