Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Running Spark in Oozie using yarn-cluster

Solved Go to solution

Running Spark in Oozie using yarn-cluster

New Contributor

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

  

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

6 REPLIES 6

Re: Running Spark in Oozie using yarn-cluster

hi,

 

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

 

Thanks !

Siva

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

Re: Running Spark in Oozie using yarn-cluster

New Contributor

Hi,

 

I am running a spark job but it fails. 

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.


@sonups3 wrote:

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works


 

Re: Running Spark in Oozie using yarn-cluster

Explorer

did you fix this issue?

Re: Running Spark in Oozie using yarn-cluster

Explorer

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

Highlighted

Re: Running Spark in Oozie using yarn-cluster

Explorer

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!