Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

Highlighted

Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

New Contributor

I'm using Hive warhouse connecteur in spark context using scalla

I want to do a Select query from Hive table that contains a column with type Array<string>

I get exception in Select 

 

User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.llap.FieldDesc.getTypeInfo()Lshadehive/org/apache/hadoop/hive/serde2/typeinfo/TypeInfo;

    at com.hortonworks.spark.sql.hive.llap.util.SchemaUtil.convertSchema(SchemaUtil.java:25)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getTableSchema(HiveWarehouseDataSourceReader.java:113)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:124)

    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.apply(DataSourceV2Relation.scala:56)

    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)

    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)

    at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:62)

    at com.orange.vodcastview.datainjector.MetaProcessorMain$.main(MetaProcessorMain.scala:34)

    at com.orange.vodcastview.datainjector.MetaProcessorMain.main(MetaProcessorMain.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)

4 REPLIES 4
Highlighted

Re: Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

New Contributor

when i test this request hive.execute("select * from test_zak") form pyspark i get this error : 

pyspark.sql.utils.IllegalArgumentException: u"Error: < expected at the end of 'array'"

Re: Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

Contributor

have you tried hive.executeQuery("select * from test_zak") instead?

Highlighted

Re: Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

New Contributor

yes, but it does'nt work

Highlighted

Re: Hive Warhouse connector don't work with array type in Hive table on HDP 3.0.1

Contributor

Can you try with this jar file instead: 

 

https://repo.hortonworks.com/content/repositories/releases/com/hortonworks/hive/hive-warehouse-conne...

 

It may not work since you're in a lower version, but its looking like a bug which could've been fixed in a recent HDP release.

Don't have an account?
Coming from Hortonworks? Activate your account here