Support Questions

Find answers, ask questions, and share your expertise

Get zookeeper quorum in Apache spark

New Contributor

Hi All,

I am new to spark , I have written a spark application which interacts with Apache phoenix on kerberised environment , and deployed on multiple cluster.

My question is how can we get zookeeper quorum in spark configuration itself to avoid manual steps. Just like retrieving properties of hdfs-site.xml,yarn-site.xml etc from sparkContext.hadoopConfiguration

Also i have seen spark sometimes read its default hbase-site.xml , leading to issue while connecting to phoenix. What are the effective ways of including HBase configuration in spark?

Spark job is invoked using yarn-client mode.

Thanks !


Super Mentor

@Poojan Kothari

If you are running Spark in Failover mode then you will find the following property to get your zookeeper URL:




export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=localhost:9983,localhost:8574,localhost:9900,localhost:9500"