Support Questions
Find answers, ask questions, and share your expertise

Spark-history reports application as incomplete even when complete?

New Contributor

Hi,

I have a 4 node HDP2.4 cluster. I submit my spark jobs on a yarn cluster.

Few of my jobs are reported as "JobSucceeded" on the terminal but are shown as incomplete applications on the 18080 spark history web UI. The spark-history logs report that the Job finished successfully as well.

Tried refreshing the page and also tried to relogin into ambari, but the spark history page still reports the job as incomplete. Read somewhere that "spark.history.yarn.diagnostics true" has to be set.. I did this as well. But it doesn't seem to help.

3 REPLIES 3

Explorer

Basic but have you tried restarting the history-server?

./sbin/start-history-server.sh restart

Guru

@Rukmini Iyer, can you please check spark history logs?

Cloudera Employee

Hi,

 

could you please look into the spark History server logs and check for any errors?

also grep for the application id in the Application History, that will show the status of the application whether it is inprogress or completed?

 

Thanks

AKR

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.