Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

clean spark-submit application left-overs from Hadoop folder?

avatar

Hello,

 

I have spark yarn client submitting jobs and when it does that, it creates a directory under my "HadoopTmp" which has files like for each submitted application:

__spark_conf__8681611713144350374.zip
__spark_libs__4985837356751625488.zip

Is there a way these can be automatically cleaned? Whenever I submit a spark job I see new entries for these again in the same folder. This is flooding up my directory what should I set to make this clear automatically? Or should i manually delete them from HDFS ?

1 REPLY 1

avatar
Master Guru

@hadoopFreak01 One idea is to set a cron job to cleanup older files periodically but I don’t think there is any inbuilt feature available to purge these files. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.