Member since
06-05-2018
10
Posts
0
Kudos Received
0
Solutions
03-03-2020
09:30 PM
1 Kudo
Could you run the administrator cmd not a normal one? From your error, I think you have not enough privilege to run these commands.
... View more
09-25-2018
10:51 AM
I'm wondering if anyone can help with my issue. I followed the blog from Eric and I can now submit jobs through oozie with spark2 on yarn. However, when I try to write to a Hive table through spark I'm getting an error. In my session, I enable hive support: sparkBuilder.enableHiveSupport() I'm then trying to run an alter table via spark.sql: transactions.sparkSession.sql(s"ALTER TABLE transactions DROP IF EXISTS PARTITION(OHREQUID_PART = ${r.getInt(0)})") If I run this through spark2-submit, it works fine, but if I run through Ozzie I get the following error: User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse;
java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse; I've copied all of the spark jars from /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ to /user/oozie/share/lib/lib_<slid>/spark2/, I've uplaoded the hive-site.xml to the same directory, also copied in the oozie-sharelib-spark.jar jar. chown'd all of the files to oozie:oozie. I also have oozie.action.sharelib.for.spark=spark2 set in my properties file. I also made sur ethe jars are showing up in oozie shareliblist spark2 It seems like a dependency collision to me, but I'm not sure which jar would be causing the issue. Thanks for any insights These are the jars I have loaded in the spark2 sharelib related to hive: hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-metastore-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-serde-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-0.23-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-common-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-shims-scheduler-1.1.0-cdh5.13.3.jar hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/hive-site.xml hdfs://nameservice1/user/oozie/share/lib/lib_20180613213413/spark2/spark-hive-exec_2.11-2.3.0.cloudera2.jar
... View more
09-19-2018
10:32 PM
Hello sparamas, Below are the steps are must follow when submitting the oozie job from oozie server. Kinit with principle and keytab [ kinit user_principle -k -t key_tab ] must use fqdn along with oozie server name in command oozie job -oozie http://machine_name@domain:11000/oozie -config xxxx -run
... View more
07-17-2018
10:42 AM
Hi, am using self signed certificates and tried to enable TLS parameters to all the services, so except Oozie. Oozie is showing some health issues, Oozie webserver cannot be communicated.
... View more