Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Multiple spark history servers - how to control address in clients' spark-defaults.conf ?

Solved Go to solution
Highlighted

Multiple spark history servers - how to control address in clients' spark-defaults.conf ?

New Contributor

Hi experts

The scenario is this:
Cluster with one Spark History Server on host Y.
Using curl I added another Spark History Server to a host X.
I also created a different config group for the new SHS, with it's own specific spark.yarn.history.address with host X
The goal is that all clients will write their logs to the default SHS on host Y and whoever needs the new SHS will have to set it manually in his spark app arguments.

Problem is, after adding the new SHS and performing "refresh configs" for spark-clients all of them are getting the new SHS address in their /etc/spark2/conf/spark-defaults.conf (spark.yarn.historyServer.address = X:18081)

Where do i set which spark.yarn.historyServer.address the clients will be using ?

Thx

Adi

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Multiple spark history servers - how to control address in clients' spark-defaults.conf ?

New Contributor

Issue resolved.
In case anyone needs to add another spark history server, so in order to control clients' config regarding spark history server address:
In Ambari >>> spark >>> config >>> spark.yarn.history.addr
Replace the variables {{spark_history_server_host}}:{{spark_history_ui_port}} with hardcoded server address and port for each config group.
Do NOT use the variables.

1 REPLY 1

Re: Multiple spark history servers - how to control address in clients' spark-defaults.conf ?

New Contributor

Issue resolved.
In case anyone needs to add another spark history server, so in order to control clients' config regarding spark history server address:
In Ambari >>> spark >>> config >>> spark.yarn.history.addr
Replace the variables {{spark_history_server_host}}:{{spark_history_ui_port}} with hardcoded server address and port for each config group.
Do NOT use the variables.

Don't have an account?
Coming from Hortonworks? Activate your account here