question Re: DELETE rows in table, how HDFS file size is impacted ? in Support Questions

DELETE rows in table, how HDFS file size is impacted ?

Mike_CHU44 — Mon, 15 Apr 2024 13:43:23 GMT

Hello.

Before deleting rows in a specific table (463,462 rows in table), HDFS file size is :

$ hadoop fs -du -s -h /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut
54.2 M 162.6 M /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut

54.2 Mb is the size of 1 file and each file is replicated 2 times so 162.6 Mb is the total size, it OK.

But after deleting more than 450,000 rows in the table (12,890 rows remaining after the DELETE), the file size didn't change at all.

Is it normal ? When new rows are added in the table, file size won't grow and HDFS will 'overwrite' older data with the new one ?

Regards

Re: DELETE rows in table, how HDFS file size is impacted ?

nramanaiah — Mon, 15 Apr 2024 18:05:46 GMT

Existing files won't be rewritten by delete query, instead deleted rows ROW__ID will be written in new delete_delta folder. Read queries will apply deleted ROW__ID on existing files to exclude the rows.

Triggering Major compaction on the table will rewrite new files merging delta & delete_delta folder.

Re: DELETE rows in table, how HDFS file size is impacted ?

Adword — Mon, 15 Apr 2024 20:08:38 GMT

Deleting rows in Hive is like hiding books in a library:

File size stays the same: HDFS doesn't erase data, just marks it hidden.

New data fills "deleted" space: New info goes on those hidden shelves first.

No immediate shrink: Resizing files is slow, so HDFS waits.

That's why the file size didn't change. It's normal HDFS behavior!

Re: DELETE rows in table, how HDFS file size is impacted ?

Mike_CHU44 — Tue, 23 Apr 2024 08:10:20 GMT

Thank you, guys, for your answers.

Re: DELETE rows in table, how HDFS file size is impacted ?

ahmadnaveed — Tue, 10 Mar 2026 12:07:52 GMT

Yes, this is normal behavior in Hive. When you delete rows, the underlying HDFS files usually don't shrink automatically because HDFS doesn't modify files in place. You typically need to run compaction or rewrite the table (like using INSERT OVERWRITE) to reclaim the space.

Re: DELETE rows in table, how HDFS file size is impacted ?

Tommike — Tue, 28 Apr 2026 07:30:46 GMT

Deleting rows in an HDFS-backed table does not immediately reduce file size because HDFS is immutable by design and individual records cannot be removed in place.

In Hive ACID tables, a DELETE does not touch the base data files at all. Instead, it writes a separate delete delta file that marks rows as logically deleted using row ID references. The physical file size on HDFS stays the same or increases because new delta files are being added. Actual size reduction only happens after a major compaction runs, which rewrites the base files by merging all deltas and physically excluding deleted rows, followed by the HDFS cleaner removing the old files.

In Apache Iceberg, deletes produce position or equality delete files written alongside existing data files, again increasing HDFS usage until a rewrite data files compaction purges the old data. In Apache Hudi Copy-On-Write, a DELETE rewrites the entire affected file immediately so size does reduce, but with heavy write amplification. In Merge-On-Read, deletes are appended as log files and compaction is still required for physical reclamation.

The bottom line is that DELETE is always append-driven at the HDFS storage layer regardless of table format, and true physical space reclamation requires compaction to run and obsolete files to be purged.