Wrong spark history redirection for finished jons

[#1]

Hi,

 I changed the spark history server in my cluster before several weeks.

I notice that when i'm clicking a finished spark job from cloudera manager, it's not directing me to the right spark history server url, while when i change the url manually i can see the finished the job.

[#2]

How did you move it? Did you remove the role form the one host and then add it to the other in CM?

[#3]

Yes

[#4]

I haven't done Spark History server specific but not had issues with others. It could be a bug. I'll see if I can reproduce.

What CM/CDH versions?

[#5]

CM 5.5.2

CDH 5.5.4

[#6]

Just thinking to delete the spark service and then add it.

Anyone who experienced the same issue?

[#7]

What value is spark.yarn.historyServer.address set to? I suspect that it did not get updated to the new Spark History server address so the RM UI is still linking to the old location? This setting is not in CM. I had to check the spark configs on a node.

To try to fix it you could de-select YARN as the YARN (MR2) Service in CM by clicking on the blue arrow pointing to the left. Save the configs and then re-select YARN. Otherwise you will need to specify it manually in the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf or History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf.

[#8]

YARN (MR2 Included) Service

Spark (Service-Wide)

YARN (MR2 Included)

• Missing required value: YARN (MR2 Included) Service

Cann't find spark.yarn.historyServer.address at any DataNode /etc/spark/conf files

[#9]

There shouldn't be any Spark components installed on the DN. Better stated, if there isn't a Spark role installed on a node it will not have the configs from CM. I have mine installed on the master and edge nodes and can see the configs there including the spark.yarn.historyServer.address setting.

It will complain that it is missing a requirement when you select it and that will go away after you re-select it.

[#10]

I have Spark Gateway on all DNs and the priephrals.

de selecting and re selecting the MR will have no affect and will not have any client configurations changes needed to deployed, also trying de selecting it ad run client configuration deploy will not work since it's alerting on the missing MR.

Is there any work around for this?