question Re: Running Spark in Oozie using yarn-cluster in Support Questions

Running Spark in Oozie using yarn-cluster

sonups3 — Fri, 16 Sep 2022 09:43:10 GMT

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

Re: Running Spark in Oozie using yarn-cluster

SivaBollineni — Mon, 02 Nov 2015 21:49:48 GMT

hi,

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

Thanks !

Siva

Re: Running Spark in Oozie using yarn-cluster

sonups3 — Wed, 04 Nov 2015 12:01:54 GMT

Thanks for the reply, i've solved this issue.

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

Re: Running Spark in Oozie using yarn-cluster

syadav — Fri, 20 Nov 2015 13:54:48 GMT

Hi,

I am running a spark job but it fails.

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.

@sonups3 wrote:
Thanks for the reply, i've solved this issue.

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.
In the jars option just specify the name of the jar. It works

Re: Running Spark in Oozie using yarn-cluster

zlpmichelle — Wed, 24 Feb 2016 19:30:38 GMT

did you fix this issue?

Re: Running Spark in Oozie using yarn-cluster

zlpmichelle — Wed, 24 Feb 2016 21:38:14 GMT

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

Re: Running Spark in Oozie using yarn-cluster

zlpmichelle — Wed, 24 Feb 2016 21:38:36 GMT

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!