question Re: how to delete all application logs from spark history + not by rotation !! in Support Questions

how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Sun, 18 Aug 2019 05:05:00 GMT

from

hdfs dfs -du -h /

we see that spark history take a lot space from HDFS

from ambari GUI

I choose spark

and then quick links

and then I get the history server page with all applications

I want to delete all applications from the page

how to do it because I not see the delete button ?

second

is it possible to delete the application that use hdfs by API or CLI ?

Re: how to delete all application logs from spark history + not by rotation !!

mlamairesse — Thu, 30 Nov 2017 02:35:31 GMT

Hi @Michael Bronson

Spark has a bunch of parameter to deal with job history rotation.
In particular :

spark.history.fs.cleaner.enabled true
spark.history.fs.cleaner.maxAge  12h
spark.history.fs.cleaner.interval 1h

Source : https://spark.apache.org/docs/latest/monitoring.html

In the example above :
- Rotation is active
- All Jobs > than 12 hours will be deleted
- Deletion happens at 1 hour intervals

Note these parameters need to be implemented on a environnement level ( not on a job level ).
They are usually placed in spark-default file.

Matthieu

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Thu, 30 Nov 2017 02:36:58 GMT

but I want to delete all application now ! not to wait for the retention

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 02:41:55 GMT

@Michael Bronson,

If you want to delete applications in spark2

hdfs dfs -rm -R /spark2-history/{app-id}

If you want to delete applications in spark1

hdfs dfs -rm -R /spark-history/{app-id}

Restart history servers after running the commands.

Thanks,

Aditya

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Thu, 30 Nov 2017 02:44:05 GMT

@Aditya thank you -

but how to delete all application that use HDFS , because in the page I see a lot of application around 1000 , so I cant delete one by one

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 02:45:40 GMT

hdfs dfs -rm -R /spark2-history/* will remove all applications

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Sun, 18 Aug 2019 05:04:52 GMT

ok , so if I want to remove it from the ambari GUI then , how to do it ( I ask because from the page I not see any delete option )

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 02:52:05 GMT

@Michael Bronson

You can use files view if you want to delete from GUI. I'm not sure if there is delete option in Spark history server.

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Thu, 30 Nov 2017 02:55:20 GMT

is it possible to print by CLI all application list so I will by grep capture the hdfs and appliaction ID and then remove it by hdfs dfs -rm -R /spark2-history//{app-id}

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 03:07:43 GMT

@Michael Bronson,

If you want to list all and delete all applications. You can simply do

hdfs dfs -rm -R /spark2-history/*

This folder will have only spark2 app logs and no other files. Hope this helps

(Or)

You can do the below. This should print all application IDs

curl http://{spark2history server url}:18080/api/v1/applications | grep "\"id\"" > a.txt
cut -d':' -f2 a.txt  | cut -d "\"" -f 2

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Thu, 30 Nov 2017 03:34:43 GMT

I try this but get errors ( what is wrong in my syntax ? ) , ( master02 is the name of park-history server )

curl -sH "X-Requested-By: ambari" -u admin:admin -i curl http://master02:8080/api/v1/applications

HTTP/1.1 404 Not Found

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 03:45:20 GMT

@Michael Bronson,

I see that you are using port 8080.

For spark history server port is 18080 by default

for spark2 history server port is 18081 by default. You can check the port in UI where you saw the applications

Re: how to delete all application logs from spark history + not by rotation !!

mike_bronson7 — Thu, 30 Nov 2017 03:57:49 GMT

ok , now I am using this:

curl -sH "X-Requested-By: ambari" -u "$API_USER"":""$API_PASSWORD" -i curl http://master02:18081/api/v1/applications

but no any output from command

what its wrong ?

Re: how to delete all application logs from spark history + not by rotation !!

asirna — Thu, 30 Nov 2017 04:03:39 GMT

@Michael Bronson

Run this command as is. No need to append headers and password. In above command you were using curl twice.

curl http://master02:18081/api/v1/applications | grep "\"id\"" > a.txt

Re: how to delete all application logs from spark history + not by rotation !!

AKR — Sun, 05 Jan 2020 13:39:27 GMT

Hi, if there are more no of files are present in spark history Server, then FS would not be working as expected. In that case, We may need to write a script to delete the old files that are more then 7 days ( or as per your requirement) from the Spark history server location . Thanks Arun