Spark-history reports application as incomplete even when complete?

[#1]

Hi,

I have a 4 node HDP2.4 cluster. I submit my spark jobs on a yarn cluster.

Few of my jobs are reported as "JobSucceeded" on the terminal but are shown as incomplete applications on the 18080 spark history web UI. The spark-history logs report that the Job finished successfully as well.

Tried refreshing the page and also tried to relogin into ambari, but the spark history page still reports the job as incomplete. Read somewhere that "spark.history.yarn.diagnostics true" has to be set.. I did this as well. But it doesn't seem to help.

[#2]

Basic but have you tried restarting the history-server?

./sbin/start-history-server.sh restart

[#3]

@Rukmini Iyer, can you please check spark history logs?

[#4]

Hi,

could you please look into the spark History server logs and check for any errors?

also grep for the application id in the Application History, that will show the status of the application whether it is inprogress or completed?

Thanks

AKR