question Re: Writing parquet file to hdfs for internal Hive table in Support Questions

Writing parquet file to hdfs for internal Hive table

Wilber — Wed, 12 Aug 2020 02:22:10 GMT

I'm trying to overwrite a parquet file in hdfs for a Hive table, but I noticed a bunch of individual parquet files with unique names like this,

table_directory /part-00199-edf111ef-0f7a-43ab-9393-8fc620195d4c-c000.snappy.parquet

How do I add a parquet file to a Hive table directory that's composed of several parquet file?

Re: Writing parquet file to hdfs for internal Hive table

stevenmatison — Wed, 12 Aug 2020 13:50:43 GMT

@Wilber My suggestion would be to not over write existing files, but write to staging location as external table. Then merge staging location data into final internal hive table with INSERT INTO final_table SELECT * FROM staging_table;

Re: Writing parquet file to hdfs for internal Hive table

Wilber — Fri, 14 Aug 2020 01:31:18 GMT

@stevenmatison - perfect; works! Thank you.

Re: Writing parquet file to hdfs for internal Hive table

stevenmatison — Fri, 14 Aug 2020 12:55:02 GMT

Please accept the solution as answer. Doin this helps complete the solution.