Hi, I have a 4 node HDP2.4 cluster. I submit my spark jobs on a yarn cluster. Few of my jobs are reported as "JobSucceeded" on the terminal but are shown as incomplete applications on the 18080 spark history web UI. The spark-history logs report that the Job finished successfully as well. Tried refreshing the page and also tried to relogin into ambari, but the spark history page still reports the job as incomplete. Read somewhere that "spark.history.yarn.diagnostics true" has to be set.. I did this as well. But it doesn't seem to help. ... View more