Created on
03-10-2020
03:21 PM
- last edited on
03-10-2020
08:52 PM
by
ask_bill_brooks
I'm using Hive warhouse connecteur in spark context using scalla
I want to do a Select query from Hive table that contains a column with type Array<string>
I get exception in Select
User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.llap.FieldDesc.getTypeInfo()Lshadehive/org/apache/hadoop/hive/serde2/typeinfo/TypeInfo;
at com.hortonworks.spark.sql.hive.llap.util.SchemaUtil.convertSchema(SchemaUtil.java:25)
at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getTableSchema(HiveWarehouseDataSourceReader.java:113)
at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:124)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.apply(DataSourceV2Relation.scala:56)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:62)
at com.orange.vodcastview.datainjector.MetaProcessorMain$.main(MetaProcessorMain.scala:34)
at com.orange.vodcastview.datainjector.MetaProcessorMain.main(MetaProcessorMain.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Created 03-11-2020 03:13 AM
when i test this request hive.execute("select * from test_zak") form pyspark i get this error :
pyspark.sql.utils.IllegalArgumentException: u"Error: < expected at the end of 'array'"
Created 03-15-2020 12:41 PM
have you tried hive.executeQuery("select * from test_zak") instead?
Created 03-15-2020 12:54 PM
yes, but it does'nt work
Created on 03-15-2020 02:42 PM - edited 03-15-2020 02:43 PM
Can you try with this jar file instead:
It may not work since you're in a lower version, but its looking like a bug which could've been fixed in a recent HDP release.