Reply
Expert Contributor
Posts: 112
Registered: ‎08-07-2017

HDFS trash not clearing the files

[ Edited ]

Hi,

 

We have enbaled trash in our hadoop cluster and trash interval is 1 day.

 

But still we see lakhs of files in trash.

 

Can you please suggest ?

 

Thanks,

Priya

Posts: 1,748
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: HDFS trash not clearing the files

What version of Cloudera Manager (CM) are you using? What is the value of CM - HDFS - Configuration - "Filesystem Trash Checkpoint Interval"? It should ideally be 1 hour, so any deleted file can get deleted in ~1d +1h after being moved to trash.
Expert Contributor
Posts: 112
Registered: ‎08-07-2017

Re: HDFS trash not clearing the files

@Harsh J,

We are using 5.9.2 version of cloudera manager.
Value for "Filesystem Trash Checkpoint Interval" is 1 day.
Expert Contributor
Posts: 112
Registered: ‎08-07-2017

Re: HDFS trash not clearing the files

@Harsh J,

We are using 5.9.2 version of cloudera manager.
I don't see Filesystem Trash Checkpoint Interval parameter in Cloudera manager. Value for Filesystem Trash Interval is 1 day.

Please suggest.

Thanks,
Priya
Posts: 1,748
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: HDFS trash not clearing the files

The property control field was introduced with a default of 1 hour in Cloudera Manager 5.11.0 onwards. See https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_rn_new_changed_features.ht...

In your current configuration, trash deletion may take upto 3 days, but should certainly happen.

If you cannot currently upgrade your Cloudera Manager, you can add the below to your HDFS - Configuration - "NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" field:

Name: fs.trash.checkpoint.interval
Value: 60

Or if in XML form:

<property>
<name>fs.trash.checkpoint.interval</name><value>60</value>
</property>
Highlighted
Expert Contributor
Posts: 112
Registered: ‎08-07-2017

Re: HDFS trash not clearing the files

@Harsh J,

Thanks for valuable inputs. So after setting checkpoint value to 1 hour, files will be deleted after 1 day +1 hour.

Thanks,
Priya
Announcements