We have a rather small cluster where we retain data for a limited time only, by automatically deleting the oldest files as a nightly job. The effect of this used to be clearly visible on the HDFS Capacity chart as a drop at the beginning of each day. After upgrading from CDH 5 beta2 to version 5.0.0-1 this is no longer the case. The files are still removed from their original locations, but we no longer seem to regain any space on HDFS, even after the trash folder is emptied.
The chart below illustrates this change in behaviour quite clearly. The upgrade happened on april 3.
An important fact in this is that on april 1 we started importing data as .gz files instead of plain text, as we were close to over-filling the disks. This has brought down the daily ingestion rate from about 700 to 130 GB/day (non-replicated). This means that during the 8 days that have passed since this change was made, we would have expected to reduce our hdfs use by some 12-15 TB. But as the graph shows, this has clearly not happened. Instead, the capacity used is actually increasing day by day.
Is there something that has changed in how hdfs deletes files, and require som additional step to get rid of data permanently? Or is it more likely that we are simply doing something wrong, or have incorrect expectations?
Running hdfs fsck / gives the following output:
Total size: 30077539630787 B
Total dirs: 9262
Total files: 311668
Total symlinks: 0
Total blocks (validated): 369784 (avg. block size 81338131 B)
Minimally replicated blocks: 369784 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9990237
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
So, as I understand it this would mean that we are using about 28*3 = 84 TB of disk, which is roughly what we would expect. (Not really sure why the avg block replication is not exactly 3 though)
This number is also consistent with what is reported by hadoop fs -du /
Now, if I instead run hdfs dfsadmin -report, I get different figures:
Configured Capacity: 125179101388800 (113.85 TB)
Present Capacity: 119138830122874 (108.36 TB)
DFS Remaining: 11580818980864 (10.53 TB)
DFS Used: 107558011142010 (97.82 TB)
DFS Used%: 90.28%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
So, DFS Used is ~98 TB, which is 14 TB more than what we are really using.
We are starting to run out of disk space on the cluster now, so any suggestions would be most welcome. I have upgraded the HDFS metadata and run the Finalized metadata upgrade command.
We are getting very close to completely filling up some of the disks now. I am not really sure what the result will be when that happens but have a feeling it will not be all good...
Here is what happens when trying to release disk space by emptying the trash:
The HDFS Used briefly drops, as the non-DFS used goes up. This is probably what would be expected to happen, only that instead of the data then being removed from the local file system, it seems to get right back into HDFS again.
There were no other file imports happening at the same time, and there are no warnings or errors in the HDFS log file.
This is becoming rather critical, so any ideas are very welcome.