question Re: Should i need to cleaning up of tmp space in hadoop cluster on weekly basis ? if yes how can i do it? please suggest in Archives of Support Questions (Read Only)

Should i need to cleaning up of tmp space in hadoop cluster on weekly basis ? if yes how can i do it? please suggest

g_rao9704 — Thu, 07 Jul 2016 17:13:19 GMT

Re: Should i need to cleaning up of tmp space in hadoop cluster on weekly basis ? if yes how can i do it? please suggest

rreddy — Fri, 08 Jul 2016 05:24:52 GMT

I'm assuming you are referring to /tmp/ directory in hdfs. You can use below command to clean it up and cron it to run every week.

hadoop fs -rm -r /tmp/*

Re: Should i need to cleaning up of tmp space in hadoop cluster on weekly basis ? if yes how can i do it? please suggest

g_rao9704 — Fri, 08 Jul 2016 13:34:08 GMT

@Rahul Reddy

Thank you so much Rahul...so if i deleted hdfs /tmp directory which is not effect my current jobs?

Re: Should i need to cleaning up of tmp space in hadoop cluster on weekly basis ? if yes how can i do it? please suggest

ledel — Sat, 09 Jul 2016 00:19:46 GMT

@sankar rao

you shouldn't wipe the entire /tmp directory, this would affect your current jobs indeed.

There's no builtin way to do that but you can cron a job which deletes the files/directories older than x days

You'll find some examples around, here is a shell (dirty but efficient) easy way for cleaning up files only:

#!/bin/bash
usage="Usage: dir_diff.sh [days]"

if [ ! "$1" ]
then
  echo $usage
  exit 1
fi

now=$(date +%s)
hadoop fs -ls -R /tmp/ | grep "^-" | while read f; do
  dir_date=`echo $f | awk '{print $6}'`
  difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))

  if [ $difference -gt $1 ]; then
    hdfs dfs -rm -f $(echo $f | awk '{print $NF}');
  fi
done