Spark Application history not found Application is still in progress

[#1]

Hi,

Since i performed the cloudera upgrade from CDH 5.3.2 to 5.4.5, i have an issue with the spark history server.

I found a similar problem on https://github.com/apache/spark/pull/4848

When application is finished running abnormally (Ctrl + c for example), the history event log file is still ends with .inprogress suffix. And the application state can not be showed on webUI, User can only see "Application history not foud xxxx, Application xxx is still in progress".

Any idea ?

regards,

[#2 · Accepted solution]

This may be too unspecific to be helpful, but I recall several JIRAs fixed for Spark 1.4 that concern the .inprogress files and history server. I expect that whatever this is could be related. If so, then the fix would be coming in 5.5 at the latest.