Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

New Contributor

I'm using Hive warhouse connecteur in spark context using scalla

I want to do a Select query from Hive table that contains a column with type Array<string>

I get exception in Select 

 

User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.llap.FieldDesc.getTypeInfo()Lshadehive/org/apache/hadoop/hive/serde2/typeinfo/TypeInfo;

    at com.hortonworks.spark.sql.hive.llap.util.SchemaUtil.convertSchema(SchemaUtil.java:25)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getTableSchema(HiveWarehouseDataSourceReader.java:113)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:124)

    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.apply(DataSourceV2Relation.scala:56)

    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)

    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:62)

    at com.orange.vodcastview.datainjector.MetaProcessorMain$.main(MetaProcessorMain.scala:34)

    at com.orange.vodcastview.datainjector.MetaProcessorMain.main(MetaProcessorMain.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

4 REPLIES 4

New Contributor

when i test this request hive.execute("select * from test_zak") form pyspark i get this error : 

pyspark.sql.utils.IllegalArgumentException: u"Error: < expected at the end of 'array'"

Contributor

have you tried hive.executeQuery("select * from test_zak") instead?

New Contributor

yes, but it does'nt work

Contributor

Can you try with this jar file instead: 

 

https://repo.hortonworks.com/content/repositories/releases/com/hortonworks/hive/hive-warehouse-conne...

 

It may not work since you're in a lower version, but its looking like a bug which could've been fixed in a recent HDP release.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.