spark2 even logs cleaning issue

[#1]

There are very old files under /spark2-history and because of this spark2 web ui is getting stuck on loading history , I have gone thorough some of the documentations where it is mentioned setting below property will solve the issue .Please find below property :

spark.history.fs.cleaner.enabled

spark.history.fs.cleaner.maxAge

spark.history.fs.cleaner.interval

would these property be same for the spark2 as well ? also I do not see these property already existing , and in spark docs default property is mentioned there .

https://spark.apache.org/docs/latest/monitoring.html

[#2]

Yes. Those three configuration are the same for Spark 1.6.3 and Spark 2.x. And you already find the solution for you; `spark.history.fs.cleaner.enabled=true`.

[#3]

Hi,

The properties mentioned in the question are correct. Also as mentioned below you can set some interval and Max age for how long the logs should be there.

spark.history.fs.cleaner.enabled=true
spark.history.fs.cleaner.interval=1d
spark.history.fs.cleaner.maxAge=5d

  Thanks

 AKR