I'm wondering if anyone can help with my issue. I followed the blog from Eric and I can now submit jobs through oozie with spark2 on yarn. However, when I try to write to a Hive table through spark I'm getting an error. In my session, I enable hive support: sparkBuilder.enableHiveSupport() I'm then trying to run an alter table via spark.sql: transactions.sparkSession.sql(s"ALTER TABLE transactions DROP IF EXISTS PARTITION(OHREQUID_PART = ${r.getInt(0)})") If I run this through spark2-submit, it works fine, but if I run through Ozzie I get the following error: User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse;
java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse; I've copied all of the spark jars from /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ to /user/oozie/share/lib/lib_<slid>/spark2/, I've uplaoded the hive-site.xml to the same directory, also copied in the oozie-sharelib-spark.jar jar. chown'd all of the files to oozie:oozie. I also have oozie.action.sharelib.for.spark=spark2 set in my properties file. I also made sur ethe jars are showing up in oozie shareliblist spark2 It seems like a dependency collision to me, but I'm not sure which jar would be causing the issue. Thanks for any insights These are the jars I have loaded in the spark2 sharelib related to hive: hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-metastore-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-serde-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-0.23-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-common-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-scheduler-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-site.xml hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/spark-hive-exec_2.11-2.3.0.cloudera2.jar
... View more
