Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Running Spark in Oozie using yarn-cluster

SOLVED Go to solution

Running Spark in Oozie using yarn-cluster

New Contributor

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

  

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

6 REPLIES 6

Re: Running Spark in Oozie using yarn-cluster

hi,

 

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

 

Thanks !

Siva

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Hi,

 

I am running a spark job but it fails. 

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.


@sonups3 wrote:

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works


 

Re: Running Spark in Oozie using yarn-cluster

Explorer

did you fix this issue?

Re: Running Spark in Oozie using yarn-cluster

Explorer

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

Highlighted

Re: Running Spark in Oozie using yarn-cluster

Explorer

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!