Get zookeeper quorum in Apache spark

[#1]

Hi All,

I am new to spark , I have written a spark application which interacts with Apache phoenix on kerberised environment , and deployed on multiple cluster.

My question is how can we get zookeeper quorum in spark configuration itself to avoid manual steps. Just like retrieving properties of hdfs-site.xml,yarn-site.xml etc from sparkContext.hadoopConfiguration

Also i have seen spark sometimes read its default hbase-site.xml , leading to issue while connecting to phoenix. What are the effective ways of including HBase configuration in spark?

Spark job is invoked using yarn-client mode.

Thanks !

[#2]

@Poojan Kothari

If you are running Spark in Failover mode then you will find the following property to get your zookeeper URL:

spark.deploy.zookeeper.url

.

https://dzone.com/articles/spark-and-zookeeper-fault

Example:

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=localhost:9983,localhost:8574,localhost:9900,localhost:9500"

.

https://spark.apache.org/docs/latest/spark-standalone.html