Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Running Spark in Oozie using yarn-cluster

avatar
Explorer

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

  

1 ACCEPTED SOLUTION

avatar
Explorer

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

View solution in original post

6 REPLIES 6

avatar

hi,

 

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

 

Thanks !

Siva

avatar
Explorer

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

avatar
New Contributor

Hi,

 

I am running a spark job but it fails. 

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.


@sonups3 wrote:

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works


 

avatar
Rising Star

did you fix this issue?

avatar
Rising Star

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

avatar
Rising Star

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!