Wrong spark history redirection for finished jons

[#1]

Hi,

 I changed the spark history server in my cluster before several weeks.

I notice that when i'm clicking a finished spark job from cloudera manager, it's not directing me to the right spark history server url, while when i change the url manually i can see the finished the job.

[#2]

Try setting the spark.yarn.historyServer.address manually as I mentioned previously.

[#3]

It is a Spark configuration setting.

"Otherwise you will need to specify it manually in the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf or History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf."

[#4]

Tried both with no success

Under /etc/spark/conf.cloudera.spark_on_yarn/spark-defaults.conf at the 2 RM i added also tried to add it using CM and still not directing to the right history server.

spark.yarn.historyServer.address=http://avvr-ahc101.lpdomain.com:18088

[#5]

When you added it through CM did it show up in the spark-default.conf?

[#6]

No, In one RM it was but the other RM was not, i changed it manually in the spark default at the 2 RMs and kept if the spark client configuration through CM

[#7]

i searched all the config files at the resource managers for spark and all of them directs the history spark server to the right one, where i can track the error of the redirection.

[#8]

Any new insights?

[#9]

Anyone who faced the same issue or have any insight how to solve this issue?

[#10]

This issue only can occur if spark-default.conf has the wrong config and getting overwritten from safety valve or it got changed manually from command line.

Maybe try deleting the role and add on a new host and do deploy client config and see what happens.

[#11]

This issue only can occur if spark-default.conf has the wrong config and getting overwritten from safety valve or it got changed manually from command line.

which servers i should check it's spark-default.conf about the this config, i checked both of resource managers and the parameters is right, and the safetly valve is empty at cloudera manager.

Maybe try deleting the role and add on a new host and do deploy client config and see what happens.

deleting the spark role means that i need to delete the oozie and hive before and i think it's not the right solution for this case.