Skip to main content
RS_11
RS_11
Bronze ⭐️⭐️⭐️
December 7, 2023
Question

Hive 3.1.3 - Empty files not generated when no rows returns

Hi

We've done a recent upgrade from Hive 1.1.0-cdh5.16.2 to Hive 3.1.3 and have noticed the difference in behaviour when no rows have been returned and file generation.

On the older version (Hive 1.1.0-cdh5.16.2) it use to create empty part files (000000_0) whereas since the upgrade this isn't the case.

The command to run this is beeline and the sql statement starts like:

Insert overwrite directory 'some/path'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'

I cannot find any definitive documentation to saw this behaviour does not exist in later versions like Hive 3.1.3

Thanks

3 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
December 7, 2023

@RS_11 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @Shmoo @cravani  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
cravani
Gold ⭐️
cravani
Gold ⭐️
December 7, 2023

@RS_11 

If I grasp your inquiry accurately, you are referring to the behavior in older CDH versions (specifically 5.x), where executing insert overwrite with a selection of 0 rows resulted in the creation of a 0-byte file named 000000_0. However, in the newer Hive version 3.1.3, you observe that no 0-byte files are generated in such scenarios.

Assuming my understanding is correct, may I ask why there is a desire to generate a 0-byte file? Having 0-byte or small files is deemed inefficient. It's noteworthy that there have been various changes in the past related to 0-byte files, as documented in the following issues:

Translate
DianaTorres
Community Manager
DianaTorres
Community Manager
December 11, 2023

@RS_11 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Translate
Regards, Diana Torres, Senior Community Moderator
Terms & ConditionsAccessibility statement