Hive table cannot be imported if the input format is avro in CDP

[#1]

Hive table cannot be imported if the input format is avro in CDP

The issue is not seen on CDH and once customer migrated to CDP they face the issue
For example:
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' |

The error we see in data-service logs is

2022-07-20T13:49:16.768Z - [sid=3c0f1a70-c6d6-404d-9fb8-11e5ac470a2a] - [rid=73eb3563-1f30-4e03-a9ca-68975967e925] - [XNIO-2 task-1] ERROR com.trifacta.dataservice.DataServiceController - [method=POST] - [url=/jdbc/schema] - Stack trace: org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.apache.hive.service.cli.HiveSQLException: MetaException(message:java.lang.UnsupportedOperationException: Storage schema reading not supported)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at com.trifacta.datasystem.jdbc.query.TableColumnsQuery.getColumns(TableColumnsQuery.java:68)
at com.trifacta.datasystem.jdbc.query.TableColumnsQuery.getSchema(TableColumnsQuery.java:42)
at com.trifacta.datasystem.connectors.jdbc.HiveJdbcSource.getSchema(HiveJdbcSource.java:98)
at com.trifacta.dataservice.ConnectionManager.lambda$getSchema$3(ConnectionManager.java:278)
at com.trifacta.dataservice.ConnectionManager.query(ConnectionManager.java:392)
at com.trifacta.dataservice.ConnectionManager.getSchema(ConnectionManager.java:272)
at com.trifacta.dataservice.JdbcController.getSchema(JdbcController.java:286)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

We even tried to set the below parameter in hive-site.xml as mentioned in the below doc, this is based on that error. but this fails even while import.
metastore.storage.schema.reader.impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader​
https://my.cloudera.com/knowledge/ERRORquotjavalangUnsupportedOperationException-Storage-schema?id=2...

we still have to fix this since every CDP customer with avro tables on all versions is going to run into this.

If you need any other information please let me know.

Expecting reply.

Advance In Thanks. 

[#2]

Hello @VeHa,

I hope you are doing great!

Could you please let me know how you have added the value for "metastore.storage.schema.reader.impl" in hive-site.xml? Is it done as below?

CM » Hive » Configuration » Safety Valve -Advanced Configuration for Hive-Site.xml

Property: metastore.storage.schema.reader.impl
Value:  org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader

 Let me know if this helps.

Cheers!

[#3]

Hi tj2007,

Thanks for reply and 

yes, its metastore.storage.schema.reader.impl" in hive-site.xml.

Thanks

[#4]

Any update on above error issue #clouera #avro..