how to delete all application logs from spark history + not by rotation !!

[#1]

from

hdfs dfs -du -h /

we see that spark history take a lot space from HDFS

from ambari GUI

I choose spark

and then quick links

and then I get the history server page with all applications

I want to delete all applications from the page

how to do it because I not see the delete button ?

second

is it possible to delete the application that use hdfs by API or CLI ?

[#2]

Hi @Michael Bronson

Spark has a bunch of parameter to deal with job history rotation.
In particular :

spark.history.fs.cleaner.enabled true
spark.history.fs.cleaner.maxAge  12h
spark.history.fs.cleaner.interval 1h

Source : https://spark.apache.org/docs/latest/monitoring.html

In the example above :
- Rotation is active
- All Jobs > than 12 hours will be deleted
- Deletion happens at 1 hour intervals

Note these parameters need to be implemented on a environnement level ( not on a job level ).
They are usually placed in spark-default file.

Matthieu

[#3]

but I want to delete all application now ! not to wait for the retention

[#4 · Accepted solution]

@Michael Bronson,

If you want to delete applications in spark2

hdfs dfs -rm -R /spark2-history/{app-id}

If you want to delete applications in spark1

hdfs dfs -rm -R /spark-history/{app-id}

Restart history servers after running the commands.

Thanks,

Aditya

[#5]

@Aditya thank you -

but how to delete all application that use HDFS , because in the page I see a lot of application around 1000 , so I cant delete one by one

[#6]

hdfs dfs -rm -R /spark2-history/* will remove all applications

[#7]

ok , so if I want to remove it from the ambari GUI then , how to do it ( I ask because from the page I not see any delete option )

[#8]

@Michael Bronson

You can use files view if you want to delete from GUI. I'm not sure if there is delete option in Spark history server.

[#9]

is it possible to print by CLI all application list so I will by grep capture the hdfs and appliaction ID and then remove it by hdfs dfs -rm -R /spark2-history//{app-id}

[#10]

@Michael Bronson,

If you want to list all and delete all applications. You can simply do

hdfs dfs -rm -R /spark2-history/*

This folder will have only spark2 app logs and no other files. Hope this helps

(Or)

You can do the below. This should print all application IDs

curl http://{spark2history server url}:18080/api/v1/applications | grep "\"id\"" > a.txt
cut -d':' -f2 a.txt  | cut -d "\"" -f 2

[#11]

I try this but get errors ( what is wrong in my syntax ? ) , ( master02 is the name of park-history server )

.

curl -sH "X-Requested-By: ambari" -u admin:admin -i curl http://master02:8080/api/v1/applications

.

HTTP/1.1 404 Not Found

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

[#12]

@Michael Bronson,

I see that you are using port 8080.

For spark history server port is 18080 by default

for spark2 history server port is 18081 by default. You can check the port in UI where you saw the applications

[#13]

ok , now I am using this:

.

curl -sH "X-Requested-By: ambari" -u "$API_USER"":""$API_PASSWORD" -i curl http://master02:18081/api/v1/applications

.

but no any output from command

what its wrong ?

[#14]

@Michael Bronson

Run this command as is. No need to append headers and password. In above command you were using curl twice.

curl http://master02:18081/api/v1/applications | grep "\"id\"" > a.txt

[#15]

Hi, if there are more no of files are present in spark history Server, then FS would not be working as expected. In that case, We may need to write a script to delete the old files that are more then 7 days ( or as per your requirement) from the Spark history server location . Thanks Arun