Wrong spark history redirection for finished jons

[#1]

Hi,

 I changed the spark history server in my cluster before several weeks.

I notice that when i'm clicking a finished spark job from cloudera manager, it's not directing me to the right spark history server url, while when i change the url manually i can see the finished the job.

[#2]

No, I meant the gateway role or if needed spark history server role where you don't see spark-default.conf having a correct config.

Gateway role is better. If you don't want to delete. Just add a new gateway role from CM->Spark->Instances and then see if then configuration comes correct.

[#3]

Hi @neerjakhattar

1- removed all instances from spark gateway, deployed configuration and then added the servers to the gatway and deployed the configuration but this didn't help.

2- checked all the luster instances and see the new spark history server in all spark-default.conf and no servers has any other configuration.

cat /etc/spark/conf/spark-defaults.conf | grep spark.yarn.historyServer.address
spark.yarn.historyServer.address=http://avvr-ahc101.lpdomain.com:18088

3- the valve CM config is empty.

Where these conf can be overwritten in some other places?

[#4]

any new insights?

[#5]

Don't know the exact cause that it was not working even i restarted the Yarn, when i had another chance to restart the Yarn, the issue was resolved.