Running Spark in Oozie using yarn-cluster

[#1]

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

[#2]

hi,

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

Thanks !

Siva

[#3 · Accepted solution]

Thanks for the reply, i've solved this issue.

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

[#4]

Hi,

I am running a spark job but it fails. 

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.

---

@sonups3 wrote:

Thanks for the reply, i've solved this issue.

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

---

[#5]

did you fix this issue?

[#6]

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

[#7]

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!