Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark-history reports application as incomplete even when complete?

Spark-history reports application as incomplete even when complete?

New Contributor

Hi,

I have a 4 node HDP2.4 cluster. I submit my spark jobs on a yarn cluster.

Few of my jobs are reported as "JobSucceeded" on the terminal but are shown as incomplete applications on the 18080 spark history web UI. The spark-history logs report that the Job finished successfully as well.

Tried refreshing the page and also tried to relogin into ambari, but the spark history page still reports the job as incomplete. Read somewhere that "spark.history.yarn.diagnostics true" has to be set.. I did this as well. But it doesn't seem to help.

3 REPLIES 3

Re: Spark-history reports application as incomplete even when complete?

New Contributor

Basic but have you tried restarting the history-server?

./sbin/start-history-server.sh restart

Re: Spark-history reports application as incomplete even when complete?

Guru

@Rukmini Iyer, can you please check spark history logs?

Re: Spark-history reports application as incomplete even when complete?

Cloudera Employee

Hi,

 

could you please look into the spark History server logs and check for any errors?

also grep for the application id in the Application History, that will show the status of the application whether it is inprogress or completed?

 

Thanks

AKR

Don't have an account?
Coming from Hortonworks? Activate your account here