Hive with Avro Integration

vamsi123 · ‎03-09-2017

I am new to Avro and any input on below clarifications is appreciated.

I got clarifications While reading Article on Avro with Hive.

1)we are not mentioning .avsc schema file here.I read in some other article where they will be mentioning .avsc file in Table properties.Is it mandatory or optional?

hive> CREATE EXTERNAL TABLE user_profile (id BIGINT, name STRING, bday STRING) STORED AS avro;

2)For the below change do I need to recreate the .avsc file each and every time?Is it one-time activity or for every step like a,b,c i need to recreate the .avsc file?

You can see that a number of operations can be allowed as a simple requirement change:

a)Adding a new column to a table (the “country” column in the 2nd file)

b)Dropping a column from a table (the “id” column in the 3nd file)

b)Renaming a column (the “birthday” column in the 4th file)

mqureshi · ‎03-10-2017

@vamsi valiveti

Starting with Hive 0.14, you don't need to specify Avro schema. It can be interpreted by Hive (Avto Serde). Adding a new column is also supported by a simple "ALTER TABLE ADD COLUMN <column name type>". More details on this link.

Changing column name is simple since it only changes the metadata.

//change column a's name to a1.

ALTER TABLE test_change CHANGE a a1 INT;

Replace column lets you drop a column but I am not sure if its Avro is supported. According to the documentation in

link above, I don't see Replace supporting Avro.