Skip to main content
Wilber
Wilber
Gold ⭐️
August 12, 2020
Solved

Writing parquet file to hdfs for internal Hive table

I'm trying to overwrite a parquet file in hdfs for a Hive table, but I noticed a bunch of individual parquet files with unique names like this, 

table_directory /part-00199-edf111ef-0f7a-43ab-9393-8fc620195d4c-c000.snappy.parquet

 

How do I add a parquet file to a Hive table directory that's composed of several parquet file?

 

 

stevenmatison
stevenmatisonAnswer
Platinum ⭐️⭐️
August 12, 2020

@Wilber My suggestion would be to not over write existing files, but write to staging location as external table.  Then merge staging location data into final internal hive table with INSERT INTO final_table SELECT * FROM staging_table;

View in thread

3 replies

stevenmatison
stevenmatisonAnswer
Platinum ⭐️⭐️
August 12, 2020

@Wilber My suggestion would be to not over write existing files, but write to staging location as external table.  Then merge staging location data into final internal hive table with INSERT INTO final_table SELECT * FROM staging_table;

Translate
    Wilber
    WilberAuthor
    Gold ⭐️
    August 14, 2020

    @stevenmatison  - perfect; works! Thank you.

    Translate
      stevenmatison
      stevenmatison
      Platinum ⭐️⭐️
      August 14, 2020

      Please accept the solution as answer.  Doin this helps complete the solution.

      Translate
        Terms & ConditionsAccessibility statement