Cloudera Community
RobXC
user rank
New Contributor

Re: Impala performance on table with array complex...

by Guru Guru in Support Questions
‎02-07-2019 09:41 AM
1 Kudo
‎02-07-2019 09:41 AM
1 Kudo
If you want to do the implicit join between the table and the nested collection, you need to reference the nested collection using the alias that you used for the table. Otherwise the top-level table and the nested collection are treated as independent table references and the query means "return the cartesian product of the tables".   I.e. you want to rewrite as follows:   select rta.transaction_purchase_id, rta.cigarette_transaction_flag, rta.non_cig_merch_transaction_flag, bow.item from wdl_atomic.retail_transaction_attribute rta, rta.retail_offering_material_group_distinct_list bow where rta.fiscal_period_id = 2019001; That will solve your issue. ... View more

Re: Spark 2 on YARN via Oozie Hue wrong version

by New Contributor in Support Questions
‎09-25-2018 10:51 AM
‎09-25-2018 10:51 AM
I'm wondering if anyone can help with my issue.  I followed the blog from Eric and I can now submit jobs through oozie with spark2 on yarn.  However, when I try to write to a Hive table through spark I'm getting an error.  In my session, I enable hive support: sparkBuilder.enableHiveSupport()   I'm then trying to run an alter table via spark.sql:  transactions.sparkSession.sql(s"ALTER TABLE transactions DROP IF EXISTS PARTITION(OHREQUID_PART = ${r.getInt(0)})")     If I run this through spark2-submit, it works fine, but if I run through Ozzie I get the following error:   User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse; java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse; I've copied all of the spark jars from /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ to /user/oozie/share/lib/lib_<slid>/spark2/, I've uplaoded the hive-site.xml to the same directory, also copied in the oozie-sharelib-spark.jar jar.  chown'd all of the files to oozie:oozie.  I also have oozie.action.sharelib.for.spark=spark2 set in my properties file.  I also made sur ethe jars are showing up in oozie shareliblist spark2   It seems like a dependency collision to me, but I'm not sure which jar would be causing the issue.   Thanks for any insights       These are the jars I have loaded in the spark2 sharelib related to hive:   hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-metastore-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-serde-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-0.23-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-common-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-scheduler-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-site.xml hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/spark-hive-exec_2.11-2.3.0.cloudera2.jar   ... View more
Contact Me
Online
Offline
Last Visited
‎06-21-2022 11:08 AM
Community Statistics
Member Since ‎08-14-2018 06:22 AM
Last Visited ‎06-21-2022 11:08 AM
Posts 3