Support Questions
Find answers, ask questions, and share your expertise

HI I have a HAdoop setup in Ubuntu16 and in that i need to clean spark history please help me to configure auto cleaning in ambari.

HI I have a HAdoop setup in Ubuntu16 and in that i need to clean spark history please help me to configure auto cleaning in ambari.

 
1 REPLY 1

Re: HI I have a HAdoop setup in Ubuntu16 and in that i need to clean spark history please help me to configure auto cleaning in ambari.

Super Mentor

@shrinivas acharya
You can set the following properties inside the spark2-defaults to Specifies whether the History Server should periodically clean up event logs from storage. For more information on the same please refer to: https://spark.apache.org/docs/latest/monitoring.html

spark.history.fs.cleaner.enabled=true
spark.history.fs.cleaner.interval=1d
spark.history.fs.cleaner.maxAge=7d



In the Ambari UI you can achieve it by the following approach:

Ambari UI --> Spark2 --> Configs (Tab) --> Advanced (Sub Tab) --> "Custom spark2-defaults" 

then click on "Add Property" then add the above properties there followed by a restart of the affected services.

.