question Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB. in Support Questions

Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

SK1 — Wed, 24 Feb 2016 21:12:35 GMT

Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

I have gone through below one but I am looking for any shell script.

https://github.com/nmilford/clean-hadoop-tmp/blob/master/clean-hadoop-tmp

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

bleonhardi — Wed, 24 Feb 2016 21:19:48 GMT

Create a file /scripts/myLogCleaner.sh ( or whatever )

add the following command ( which deletes all files having a log in the name and are older than a day )

find /tmp/hive -name *log* -mtime +1 -exec rm {} \;

and crontab it.

crontab -e

0 0 * * * /scripts/myLogCleaner.sh

This will start the cleaner every day at midnight.

( obviously just one out of approximately 3 million different ways to do it 🙂 )

Edit: ah not the logs of the hive CLI but the scratch dir of hive. That makes it a bit harder since there is no hadoop find. Weird that it grows so big it should clean up after itself unless the command line interface or task gets killed.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

aervits — Wed, 24 Feb 2016 21:23:22 GMT

this is on hdfs Benjamin. I mean same approach just hdfs commands vs local fs.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

SK1 — Wed, 24 Feb 2016 21:37:24 GMT

@Benjamin Leonhardi : I can do it easily with local but I am looking for hdfs /tmp/hive dir.

So do we have anything like this for hdfs.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

bleonhardi — Wed, 24 Feb 2016 21:43:01 GMT

That would be the time when I start writing some python magic parsing the timestamp from the hadoop -ls output command. Or to be faster a small Java program doing the same with the FileSystem API.

Someone already did the first approach with shell script apparently. Replace the echo with a hadoop fs -rm -r -f and you might be good. But I didn't test it obviously ...

http://stackoverflow.com/questions/12613848/finding-directories-older-than-n-days-in-hdfs

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

aervits — Wed, 24 Feb 2016 21:44:26 GMT

@Benjamin Leonhardi yep, I've done that a while ago with java hdfs api. Look up the paths, identify age of files, delete.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

rahulpathak109 — Wed, 24 Feb 2016 21:54:09 GMT

@Saurabh Kumar

I have not used this but worth trying.

https://issues.apache.org/jira/browse/HADOOP-8989

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

iroberts — Fri, 08 Apr 2016 22:05:22 GMT

@Saurabh Kumar To add to this, you could investigate third party dev projects such as https://github.com/nmilford/clean-hadoop-tmp

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

trainings — Wed, 31 Aug 2016 06:24:57 GMT

You can do:

#!/bin/bash usage="Usage: dir_diff.sh [days]" if [ ! "$1" ] then echo $usage exit 1 fi now=$(date +%s) hadoop fs -ls -R /tmp/ | grep "^d" | while read f; do dir_date=`echo $f | awk '{print $6}'` difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) )) if [ $difference -gt $1 ]; then hadoop fs -rm -r `echo $f | awk '{ print $8 }'`; fi done

Replace the directories or files you need to clean up appropriately.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

SK1 — Wed, 21 Sep 2016 17:16:43 GMT

@Gurmukh Singh: I tried this script and not getting anything just below output.

[user@server2~]$ ./cleanup.sh

Usage: dir_diff.sh [30]

I have same thing in script which you have mentioned.

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

trainings — Thu, 22 Sep 2016 19:51:55 GMT

@Saurabh the script takes a argument as the number of days 🙂

So, if you want to look for files older then 10 days then #./cleaup.sh 10

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

SK1 — Sat, 24 Sep 2016 14:18:49 GMT

@Gurmukh Singh: Thanks I just tested it with following ways and it is working fine. We can change hadoop fs -ls to hadoop fs -rm -r and required dir.

#!/bin/bash

usage="Usage: dir_diff.sh [days]"

if [!"$1"]

then

echo$usage

exit1

now=$(date +%s)

hadoop fs -ls /zone_encr2/ | grep "^d" | while read f; do

dir_date=`echo $f | awk '{print $6}'`

difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))

if [$difference-gt$1]; then

hadoop fs -ls `echo$f| awk '{ print $8 }'`;

done

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

trainings — Tue, 11 Oct 2016 13:52:30 GMT

@SaurabhSaurabh

Yes, the script I gave was with "hadoop fs -ls" command, because many people do not understand what it does and they will simply copy the script, run it and then blame that they lost data.

The problem is most people, call themselves Hadoop admins, but have never worked as Linux system admins/engineer 🙂

Re: Do we have any script which we can use to clean /tmp/hive/ dir frequently on hdfs. Because it is consuming space in TB.

SY0C64110 — Thu, 11 Apr 2019 22:27:10 GMT

can some help me on this as well?

https://community.hortonworks.com/questions/243908/major-compaction-failure.html