Create Hive ORC table from avro file

younes_kafi — Tue, 08 Nov 2016 20:39:30 GMT

Hello,

Is it possible to create an internal ORC hive table from an avro hdfs file?

I tried something like this:

CREATE TABLE orc_table
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED as orc
LOCATION '/user/someuser/avro_folder/'
TBLPROPERTIES ('avro.schema.url'='hdfs:///user/someuser/schema.avsc');

but I get this error:

Failed with exception java.io.IOException:java.lang.RuntimeException: serious problem

Beside a describe formatted command returns:

# Storage Information
SerDe Library:          org.apache.hadoop.hive.serde2.avro.AvroSerDe
InputFormat:            org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat:           org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

Thanks in advance.

Re: Create Hive ORC table from avro file

ssubhas — Tue, 08 Nov 2016 20:52:27 GMT

@younes kafi

The create table is incorrect, it is not possible to store the data in avro format into orc format. The best possible way is as below:

CREATE TABLE avro_table
ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
LOCATION '/user/someuser/avro_folder/'
TBLPROPERTIES ('avro.schema.url'='hdfs:///user/someuser/schema.avsc');

Create orc file as below:
CREATE TABLE orc_table stored as orc;
INSERT INTO TABLE orc_table SELECT * FROM avro_table;

question Re: Create Hive ORC table from avro file in Archives of Support Questions (Read Only)

Create Hive ORC table from avro file

Re: Create Hive ORC table from avro file