Support Questions
Find answers, ask questions, and share your expertise

spark2 even logs cleaning issue

spark2 even logs cleaning issue

Explorer

There are very old files under /spark2-history and because of this spark2 web ui is getting stuck on loading history , I have gone thorough some of the documentations where it is mentioned setting below property will solve the issue .Please find below property :

spark.history.fs.cleaner.enabled
spark.history.fs.cleaner.maxAge

spark.history.fs.cleaner.interval

would these property be same for the spark2 as well ? also I do not see these property already existing , and in spark docs default property is mentioned there .

https://spark.apache.org/docs/latest/monitoring.html

2 REPLIES 2
Highlighted

Re: spark2 even logs cleaning issue

Expert Contributor

Yes. Those three configuration are the same for Spark 1.6.3 and Spark 2.x. And you already find the solution for you; `spark.history.fs.cleaner.enabled=true`.

Highlighted

Re: spark2 even logs cleaning issue

Cloudera Employee

Hi,

The properties mentioned in the question are correct. Also as mentioned below you can set some interval and Max age for how long the logs should be there.

spark.history.fs.cleaner.enabled=true
spark.history.fs.cleaner.interval=1d
spark.history.fs.cleaner.maxAge=5d

  Thanks

 AKR