question Re: Delete application logs from spark history specifying date in Support Questions

Delete application logs from spark history specifying date

Emanuel_MXN — Thu, 21 Sep 2023 14:24:45 GMT

from

hdfs dfs -du -h /

we see that spark history take a lot space from HDFS

I want delete only some that I specified for example all that are of the year 2019 and 2019, and keep the rest.

If I use the command
hdfs dfs -rm -R /spark2-history/*

this delete all and I want not delete all.

Thanks

Re: Delete application logs from spark history specifying date

RangaReddy — Fri, 15 Sep 2023 01:32:05 GMT

Hi @Emanuel_MXN

Generally not recommended to keep event logs older than some days/months. In your case, you are keeping logs for years.

To avoid keeping the old logs, please add the following parameters to the spark-defaults.conf file and delete old event logs based on your need.

spark.history.fs.cleaner.enabled true spark.history.fs.cleaner.maxAge 7d spark.history.fs.cleaner.interval 1h

I don't have any handy script to delete files from hdfs for specific date/year. If i found definetely i will share it here.

Re: Delete application logs from spark history specifying date

VidyaSargur — Tue, 26 Sep 2023 09:42:32 GMT

@Emanuel_MXN, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.