Reply
New Contributor
Posts: 1
Registered: ‎12-23-2015

When creating a Hive table I am encountering SemanticException [Error 10043] erro

I am using Hive, CDH-5.4.7-1.cdh5.4.7.p0.3, to create a Parquet format table using literal Avro schema as follows


hive> create external table if not exists datamodel.bcdfsegway_Test1
> STORED AS PARQUET
> LOCATION 'hdfs://nameservice1/project/segway/'
> TBLPROPERTIES ('avro.schema.literal'='{
> "type" : "record",
> "name" : "Test1",
> "namespace" : "com.company.avro.generated",
> "doc" : "some comments.",
> "fields" : [ {
> "name" : "executionInfoDetails",
> "type" : [ "null", "string" ]
> }, {
> "name" : "batchTimestamp",
> "type" : [ "null", "int" ]
> }, {
> "name" : "recordTimestamp",
> "type" : [ "null", "int" ]
> }]
> }')
> ;
FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified
hive>

 

Is this the correct syntax for my purpose?

 

thanks,

Bruce 

Explorer
Posts: 21
Registered: ‎09-09-2015

Re: When creating a Hive table I am encountering SemanticException [Error 10043] erro

[ Edited ]

list of columns are required for parquet file format. One can't use AVRO schema for defining columns of a parquet table. You can try below.

 

1) specify list of columns of your external table.

2) create an avro table with your AVRO schema and then use CTAS for your Parquet table creation.

 

Cloudera Team,

 

Please guide in creating a custom serializer or using any available serializer to create a Parquet table based on avro schema.

OR

If there is any other best possible way.

 

Best regards,

Announcements