Deleting History of Spark run applications

[#1]

Guys,

I want to cleanup all the history of the applications run on Spark. Because of some issues previously still some of the applications started are being shown "incomplete" and when I check the /spark-history directory I see those applications with .in progress extension for example : /spark-history/application_14653124594105_0007.inprogress.

When I try killing it using yarn application kill command it shows that

Application application_14653124594105_0007 has already finished.

I delete the appname.inprogress directory from /spark-history but it doesn't stop appearing from the UI and when I click the link on the UI, it shows error saying : Application not found.

What is the safe way to delete the history and want to avoid displaying on the History servers.

Many thanks.

[#2 · Accepted solution]

@Smart Solutions

It kept it in memory therefore you have to restart the Spark history server after deleting it from /spark-history/ location.

[#3]

Yeah I found that after posting this 🙂 Thanks @Jitendra Yadav

[#4]

I have the same pb but i don't know how to delete my application from spark-history. Could you explain, please ?

thanks