Reply
Highlighted
Explorer
Posts: 23
Registered: ‎08-20-2015

CREATE TABLE Error When Literal Avro Schema Is Larger Than 4K

Hi.  I'm posting this because I didn't find a lot of help on Google so maybe this will help others if they run across it.  Moreover, could very well be this is fixed in a future release of Hive, so might end up being dated at some point - if not already.  But in any sense, wanted to at least share.  We are running Hive 0.13.1-cdh5.3.2.

 

We have some very large schemas we are ingesting to Hive using the Avro file format.  On the 'larger' ones, we were getting the following error when trying to create a table.

 

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Put request failed : INSERT INTO TABLE_PARAMS (PARAM_VALUE,TBL_ID,PARAM_KEY) VALUES (?,?,?) NestedThrowables:
org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO TABLE_PARAMS (PARAM_VALUE,TBL_ID,PARAM_KEY) VALUES (?,?,?) ) (state=08S01,code=1)

 

There wasn't much here, but we did stumble upon this JIRA that while I can't tell 100% if it's a similar issue we have, it at least got us pointed in the right direction.  

 

https://issues.cloudera.org/browse/KITE-469

 

In any sense, when using the 'avro.schema.literal' name/value pair in Hive's TBLPROPERTIES, we get this error when the avro schema is larger than 4K.  In other words, in the example below, if that is more than 4K, you'll get the error.

 

The fix for this on our end was to simply make the Avro a separate file in HDFS and reference it via URL.  The second example below is how we fixed it.

 

Again, no action needed.  Just posting for reference.

 

--errors

CREATE EXTERNAL TEST_TABLE_1
(
COL1 STRING
)
STORED AS AVRO
LOCATION '/archive/TEST_TABLE_1'
TBLPROPERTIES ('avro.schema.literal'='{<4K of data>}');

 

--works

CREATE EXTERNAL TEST_TABLE_1
(
COL1 STRING
)
STORED AS AVRO
LOCATION '/archive/TEST_TABLE_1'
TBLPROPERTIES ('avro.schema.url'='/archive/avro_schemas/TEST_TABLE_1.avsc');

 

Announcements