Created 08-08-2017 02:38 PM
Trying to run a spark job from oozie. Below is the code which I am trying to run.
SparkConf conf = getConf(appName);
JavaSparkContext sc = new JavaSparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);
I am getting the following error:
JOB[0000000-170808082825775-oozie-oozi-W] ACTION[0000000-170808082825775-oozie-oozi-W@Sample-node] Launcher exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
Here's my workflow xml file
<workflow-app name="DataSampling" xmlns="uri:oozie:workflow:0.4">
<start to='Sample-node'/>
<action name="Sample-node">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>/hdp/apps/2.5.3.0-37/tez/tez.tar.gz</value>
</property>
</configuration>
<master>${master}</master>
<mode>${mode}</mode>
<name>Sample class on Oozie - Sampling</name>
<class>Sampling</class>
<jar>/path/jarfile.jar</jar>
<arg>${numEventsPerPattern}</arg>
<arg>${eventdate}</arg>
<arg>${eventtype}</arg>
<arg>${user}</arg>
</spark>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Workflow failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
</message>
</kill>
<end name='end'/>
</workflow-app>
I am using Hortonworks Data Platform 2.5. Can any one please help if I am missing some thing in the classpath.
Thanks in advance.
Created 08-08-2017 02:48 PM
error log is
Caused by: org.datanucleus.api.jdo.exceptions.ClassNotPersistenceCapableException: The class "org.apache.hadoop.hive.metastore.model.MDatabase" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found. NestedThrowables: org.datanucleus.exceptions.ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MDatabase" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found. at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:473) at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:720) at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:740) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:521) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) at com.sun.proxy.$Proxy36.createDatabase(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:604) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) ... 50 more Caused by: org.datanucleus.exceptions.ClassNotPersistableException: The class "org.apache.hadoop.hive.metastore.model.MDatabase" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found. at org.datanucleus.ExecutionContextImpl.assertClassPersistable(ExecutionContextImpl.java:5518) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:1979) at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1922) at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1777) at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217) at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:715) ... 66 more
Created 08-08-2017 03:15 PM
Looks like Spark/Oozie is not able to find the hive site file and thus not able to create a HiveContext.
Follow this KB article to set up Oozie Spark Action to talk to Hive.
https://community.hortonworks.com/articles/88288/spark-action-to-access-hive-using-hivecontext.html
Created 08-08-2017 05:28 PM
Thanks for replay Dinesh Chitlangia
I already added hive-site file in /user/oozie/share/lib/lib201650546/spark directory.
Created 08-09-2017 12:05 PM
Issue has been resolved... i followed this link
https://community.hortonworks.com/questions/25847/hive-metastore-method-not-found.html
Thanks for support