DELETE rows in table, how HDFS file size is impacted ?

Existing files won't be rewritten by delete query, instead deleted rows ROW__ID will be written in new delete_delta folder. Read queries will apply deleted ROW__ID on existing files to exclude the rows.

Triggering Major compaction on the table will rewrite new files merging delta & delete_delta folder.

Thank you, guys, for your answers.

Deleting rows in an HDFS-backed table does not immediately reduce file size because HDFS is immutable by design and individual records cannot be removed in place.

In Hive ACID tables, a DELETE does not touch the base data files at all. Instead, it writes a separate delete delta file that marks rows as logically deleted using row ID references. The physical file size on HDFS stays the same or increases because new delta files are being added. Actual size reduction only happens after a major compaction runs, which rewrites the base files by merging all deltas and physically excluding deleted rows, followed by the HDFS cleaner removing the old files.

In Apache Iceberg, deletes produce position or equality delete files written alongside existing data files, again increasing HDFS usage until a rewrite data files compaction purges the old data. In Apache Hudi Copy-On-Write, a DELETE rewrites the entire affected file immediately so size does reduce, but with heavy write amplification. In Merge-On-Read, deletes are appended as log files and compaction is still required for physical reclamation.

The bottom line is that DELETE is always append-driven at the HDFS storage layer regardless of table format, and true physical space reclamation requires compaction to run and obsolete files to be purged.

This behavior is expected if the table is transactional (ACID-enabled).

A DELETE operation does not immediately rewrite the underlying HDFS data files. Instead, Hive records the deleted row identifiers in a delete_delta directory, and query engines apply those delete markers when reading the table. As a result, the original data files remain in place and the HDFS size often stays the same immediately after a large delete.

If you deleted 450,000+ rows and only have ~13,000 rows remaining, it's normal that the table directory still occupies roughly the same amount of space. In some cases, storage consumption can even increase temporarily because the delete metadata itself must be stored.

To actually reclaim disk space, you typically need to run a major compaction. During major compaction, Hive rewrites the data files, merges delta/delete_delta information, and removes data that is no longer visible to queries. Only after that process completes will you generally see a significant reduction in HDFS usage.

One additional point: new inserts do not "overwrite" the deleted rows inside the existing files. HDFS files are immutable, so Hive creates new data files rather than modifying existing ones in place. The cleanup and consolidation happen during compaction rather than during the DELETE itself.

You may want to check:

• Whether the table is ACID/transactional.

• The contents of the delta_* and delete_delta_* directories.

• When the next automatic major compaction is scheduled, or whether a manual major compaction is appropriate for your environment.

This is a great explanation of how Hive and HDFS handle deletes. Good documentation also makes a big difference here. Using clear typography, consistent formatting, and readable text for concepts like delete_delta, compaction, and immutable files makes technical discussions much easier to understand, especially for developers who are new to Hive internals.

Deleting Hive rows is more like marking pages as unused:

Data isn't immediately removed: Hive can leave the underlying files unchanged after a DELETE.

Space can be reused: New records may eventually occupy the space associated with older deleted data.

File size stays unchanged: HDFS generally doesn't shrink files just because rows were deleted.

Cleanup happens separately: Compaction or other maintenance processes may be needed to physically reduce storage usage.

That’s why the file size can remain the same after deleting rows—it’s expected behavior with Hive and HDFS.