spark-defaults.conf constantly being overwritten

[#1]

I added a jar to my classpath by adding the extraClassPath options in spark-defaults.conf. Specifically, these two lines:

spark.executor.extraClassPath /usr/hdp/2.4.2.0-258/phoenix/phoenix-4.4.0.2.4.2.0-258-client-spark.jar:/usr/share/java/sqljdbc41.jar
spark.driver.extraClassPath   /usr/hdp/2.4.2.0-258/phoenix/phoenix-4.4.0.2.4.2.0-258-client-spark.jar:/usr/share/java/sqljdbc41.jar

I wasn't sure which spark-defaults.conf to edit (here are all of them):

$ find / -name spark-defaults*.conf* 2> /dev/null
/usr/hdp/2.4.2.0-258/etc/spark/conf/spark-defaults.conf.template
/usr/hdp/2.4.2.0-258/etc/spark/conf/spark-defaults.conf
/etc/spark/conf.backup/spark-defaults.conf.template
/etc/spark/conf.backup/spark-defaults.conf
/etc/spark/2.4.2.0-258/0/spark-defaults.conf.template
/etc/spark/2.4.2.0-258/0/spark-defaults.conf

After some trial and error, my jar only ran successfully when I added the extraClassPath options to /etc/spark/2.4.2.0-258/0/spark-defaults.conf. Editing the others did not resolve the "class not found" exceptions.

However, it seems that my edits to this file are constantly being erased/overwritten. When I tried to run the jar this morning, the lines I had added to the spark-defaults.conf were no longer there, and at the top of the file was "# Generated by Apache Ambari. Mon Jul 4 00:06:17 2016". When I re-added the two lines, everything worked as it did yesterday.

Is there another file I should be editing instead of (or in addition to) /etc/spark/2.4.2.0-258/0/spark-defaults.conf?

[#2 · Accepted solution]

Hi @Zach Kirsch,

I think you are adding your custom jars via editing spark-defaults.conf from command line. Ambari will overwrite your changes when you restart the services(please note that manual editing is not compatible when you manage cluster with ambari).

Instead, go to Ambari UI --> select Spark service --> select config window --> find "Custom spark-defaults" section and add your jars there. pls refer below sample screenshot:

Thanks,

Jobin

[#3]

I launch my cluster from the command line (via Azure and an ARM template), and ideally I want it to be set up automatically with the right classpath. Is there a way of automating this process (rather than going through the Ambari UI)?

[#4]

Hi @Zach Kirsch,

I am assuming you need to update it from command line: you have two options:

- Use Ambari Api and do it

- Use Ambari configs.sh and get it done. (this one will be easy for you i guess)

example:

/var/lib/ambari-server/resources/scripts/configs.sh set node1 Hortonworks spark-defaults "spark.history.ui.port" "18090"

[Note: "node1" is my server, "Hortonworks" is my cluster name. Also you will have to restart service to get changes to take effect]

you can find more details here:

https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

Thanks,

Jobin

[#5]

Thank you!