Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Get zookeeper quorum in Apache spark

New Contributor

Hi All,

I am new to spark , I have written a spark application which interacts with Apache phoenix on kerberised environment , and deployed on multiple cluster.

My question is how can we get zookeeper quorum in spark configuration itself to avoid manual steps. Just like retrieving properties of hdfs-site.xml,yarn-site.xml etc from sparkContext.hadoopConfiguration

Also i have seen spark sometimes read its default hbase-site.xml , leading to issue while connecting to phoenix. What are the effective ways of including HBase configuration in spark?

Spark job is invoked using yarn-client mode.

Thanks !


Super Mentor

@Poojan Kothari

If you are running Spark in Failover mode then you will find the following property to get your zookeeper URL:




export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=localhost:9983,localhost:8574,localhost:9900,localhost:9500"


Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.