Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Deleting files does not clear disk space

Deleting files does not clear disk space

Contributor

Hi,

 

We have a rather small cluster where we retain data for a limited time only, by automatically deleting the oldest files as a nightly job. The effect of this used to be clearly visible on the HDFS Capacity chart as a drop at the beginning of each day. After upgrading from CDH 5 beta2 to version 5.0.0-1 this is no longer the case. The files are still removed from their original locations, but we no longer seem to regain any space on HDFS, even after the trash folder is emptied.

 

The chart below illustrates this change in behaviour quite clearly. The upgrade happened on april 3.

 

hdfs.png

 

An important fact in this is that on april 1 we started importing data as .gz files instead of plain text, as we were close to over-filling the disks. This has brought down the daily ingestion rate from about 700 to 130 GB/day (non-replicated). This means that during the 8 days that have passed since this change was made, we would have expected to reduce our hdfs use by some 12-15 TB. But as the graph shows, this has clearly not happened. Instead, the capacity used is actually increasing day by day.

 

Is there something that has changed in how hdfs deletes files, and require som additional step to get rid of data permanently? Or is it more likely that we are simply doing something wrong, or have incorrect expectations?

 

Best regards

\Knut

 

EDIT:

 

Running hdfs fsck / gives the following output:

 

Status: HEALTHY
Total size: 30077539630787 B
Total dirs: 9262
Total files: 311668
Total symlinks: 0
Total blocks (validated): 369784 (avg. block size 81338131 B)
Minimally replicated blocks: 369784 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9990237
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1

 

So, as I understand it this would mean that we are using about 28*3 = 84 TB of disk, which is roughly what we would expect. (Not really sure why the avg block replication is not exactly 3 though)

 

This number is also consistent with what is reported by hadoop fs -du /

 

 

0 /system
1409345170004 /tmp
28668194460783 /user

 

Now, if I instead run hdfs dfsadmin -report, I get different figures:


Configured Capacity: 125179101388800 (113.85 TB)
Present Capacity: 119138830122874 (108.36 TB)
DFS Remaining: 11580818980864 (10.53 TB)
DFS Used: 107558011142010 (97.82 TB)
DFS Used%: 90.28%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

 

So, DFS Used is ~98 TB, which is 14 TB more than what we are really using.

 

We are starting to run out of disk space on the cluster now, so any suggestions would be most welcome. I have upgraded the HDFS metadata and run the Finalized metadata upgrade command. 

3 REPLIES 3

Re: Deleting files does not clear disk space

Contributor

UPDATE:

 

We are getting very close to completely filling up some of the disks now. I am not really sure what the result will be when that happens but have a feeling it will not be all good...

 

Here is what happens when trying to release disk space by emptying the trash:

 

hdfs_delete.png

The HDFS Used briefly drops, as the non-DFS used goes up. This is probably what would be expected to happen, only that instead of the data then being removed from the local file system, it seems to get right back into HDFS again.

 

There were no other file imports happening at the same time, and there are no warnings or errors in the HDFS log file.

 

This is becoming rather critical, so any ideas are very welcome.

Highlighted

Re: Deleting files does not clear disk space

New Contributor

Hi,

 

Any updates on this.  What was the underlying issue and how was it resolved?

Re: Deleting files does not clear disk space

Contributor