question Re: Deleting History of Spark run applications in Archives of Support Questions (Read Only)

Deleting History of Spark run applications

smartninja723 — Thu, 09 Jun 2016 16:16:33 GMT

Guys,

I want to cleanup all the history of the applications run on Spark. Because of some issues previously still some of the applications started are being shown "incomplete" and when I check the /spark-history directory I see those applications with .in progress extension for example : /spark-history/application_14653124594105_0007.inprogress.

When I try killing it using yarn application kill command it shows that

Application application_14653124594105_0007 has already finished.

I delete the appname.inprogress directory from /spark-history but it doesn't stop appearing from the UI and when I click the link on the UI, it shows error saying : Application not found.

What is the safe way to delete the history and want to avoid displaying on the History servers.

Many thanks.

Re: Deleting History of Spark run applications

jyadav — Thu, 09 Jun 2016 16:31:02 GMT

@Smart Solutions

It kept it in memory therefore you have to restart the Spark history server after deleting it from /spark-history/ location.

Re: Deleting History of Spark run applications

smartninja723 — Thu, 09 Jun 2016 17:22:47 GMT

Yeah I found that after posting this 🙂 Thanks @Jitendra Yadav

Re: Deleting History of Spark run applications

max_perelmuter — Tue, 20 Dec 2016 19:37:15 GMT

I have the same pb but i don't know how to delete my application from spark-history. Could you explain, please ?

thanks