Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark-history reports application as incomplete even when complete?

avatar
New Contributor

Hi,

I have a 4 node HDP2.4 cluster. I submit my spark jobs on a yarn cluster.

Few of my jobs are reported as "JobSucceeded" on the terminal but are shown as incomplete applications on the 18080 spark history web UI. The spark-history logs report that the Job finished successfully as well.

Tried refreshing the page and also tried to relogin into ambari, but the spark history page still reports the job as incomplete. Read somewhere that "spark.history.yarn.diagnostics true" has to be set.. I did this as well. But it doesn't seem to help.

3 REPLIES 3

avatar
Contributor

Basic but have you tried restarting the history-server?

./sbin/start-history-server.sh restart

avatar
Guru

@Rukmini Iyer, can you please check spark history logs?

avatar
Cloudera Employee

Hi,

 

could you please look into the spark History server logs and check for any errors?

also grep for the application id in the Application History, that will show the status of the application whether it is inprogress or completed?

 

Thanks

AKR