Reply
Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Wrong spark history redirection for finished jons

Try setting the spark.yarn.historyServer.address manually as I mentioned previously.
Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Wrong spark history redirection for finished jons

It is a Spark configuration setting.

"Otherwise you will need to specify it manually in the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf or History Server Advanced Configuration Snippet (Safety Valve) for spark-history-server.conf."
Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

Tried both with no success

 

Under /etc/spark/conf.cloudera.spark_on_yarn/spark-defaults.conf at the 2 RM i added also tried to add it using CM and still not directing to the right history server.

 

spark.yarn.historyServer.address=http://avvr-ahc101.lpdomain.com:18088

Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Wrong spark history redirection for finished jons

When you added it through CM did it show up in the spark-default.conf?
Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

No, In one RM it was but the other RM was not, i changed it manually in the spark default at the 2 RMs and kept if the spark client configuration through CM

Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

i searched all the config files at the resource managers for spark and all of them directs the history spark server to the right one, where i can track the error of the redirection.

 

Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

Any new insights?

Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

Anyone who faced the same issue or have any insight how to solve this issue?

Cloudera Employee
Posts: 25
Registered: ‎08-22-2014

Re: Wrong spark history redirection for finished jons

This issue only can occur if spark-default.conf has the wrong config and getting overwritten from safety valve or it got changed manually from command line.

 

Maybe try deleting the role and add on a new host and do deploy client config and see what happens.

 

 

Expert Contributor
Posts: 263
Registered: ‎01-25-2017

Re: Wrong spark history redirection for finished jons

This issue only can occur if spark-default.conf has the wrong config and getting overwritten from safety valve or it got changed manually from command line.

 

which servers i should check it's spark-default.conf about the this config, i checked both of resource managers and the parameters is right, and the safetly valve is empty at cloudera manager.

 

Maybe try deleting the role and add on a new host and do deploy client config and see what happens.

 

deleting the spark role means that i need to delete the oozie and hive before and i think it's not the right solution for this case.

Announcements