Support Questions
Find answers, ask questions, and share your expertise

Querying Hive External Table on an Avro File being appended to

New Contributor

Having created a Hive External table on an Avro file in HDFS, will Hive queries return fresh data being appended to the same Avro file (partition)?

1 REPLY 1

Contributor

Hi,

 

You can append new data to an Avro-based table, for example through hive/beeline, but that will result in a new file.

 

Avro is a binary file format, so it is not so trivial to append data to the same file. If you simply append new data to an Avro file, you can break it's structure. Though there is a custom Writer for accomplishing this, see: https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/file/DataFileWriter.html

It is not relevant whether the table is external or managed, Hive queries will return the actual data stored in HDFS.

 

Hope this helps,

Cheers,

 Miklos