question Re: How to do a cleanup of hdfs files older than a certain date using a bash script in Archives of Support Questions (Read Only)

How to do a cleanup of hdfs files older than a certain date using a bash script

Former Member — Fri, 31 Mar 2017 04:07:10 GMT

How to do a cleanup of hdfs files older than a certain date using a bash script.

I am just looking for a general strategy.

Re: How to do a cleanup of hdfs files older than a certain date using a bash script

namaheshwari — Fri, 31 Mar 2017 04:26:08 GMT

Below post has one example script which deletes files older than certain days:

https://community.hortonworks.com/questions/19204/do-we-have-any-script-which-we-can-use-to-clean-tm.html

#!/bin/bash
usage="Usage: dir_diff.sh [days]"
if [!"$1"]
then
echo$usage
exit1
fi
now=$(date +%s)
hadoop fs -ls /zone_encr2/ | grep "^d" | while read f; do
dir_date=`echo $f | awk '{print $6}'`
difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))
if [$difference-gt$1]; then
hadoop fs -ls `echo$f| awk '{ print $8 }'`;
fi
done

Re: How to do a cleanup of hdfs files older than a certain date using a bash script

cesar_rodriguez — Wed, 28 Sep 2022 14:05:06 GMT

The script in the accepted solution was not working for me, so I modified it:

#!/bin/bash usage="Usage: dir_diff.sh [path] [-gt|-lt] [days]" if (( $# < 3 )) then echo $usage exit 1 fi now=$(date +%s) hdfs dfs -ls $1 | grep -v "^d" | grep -v '^Found ' | while read f; do dir_date=`echo $f | awk '{print $6}'` difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) )) if [ $difference $2 $3 ]; then echo $f # hdfs dfs -ls `echo $f| awk '{ print $8 }'`; fi done