Reply
Highlighted
New Contributor
Posts: 5
Registered: ‎10-06-2015
Accepted Solution

Running Spark in Oozie using yarn-cluster

[ Edited ]

Getting the below error on submitting spark wordcount job from hue oozie. Please help to resolve this, Thanks

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, mbda3node06.it.three.co.uk): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

  

Explorer
Posts: 7
Registered: ‎05-05-2015

Re: Running Spark in Oozie using yarn-cluster

hi,

 

I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml.

 

Thanks !

Siva

New Contributor
Posts: 5
Registered: ‎10-06-2015

Re: Running Spark in Oozie using yarn-cluster

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works

New Contributor
Posts: 2
Registered: ‎11-20-2015

Re: Running Spark in Oozie using yarn-cluster

Hi,

 

I am running a spark job but it fails. 

I have added the oozie-libpath as well.

Still it is unable to locate my jar.

Could you please help.


@sonups3 wrote:

Thanks for the reply, i've solved this issue.

 

In the workflow setting of Hue Oozie, under variables add oozie-libpath and value is the full path of the jar.

In the jars option just specify the name of the jar. It works


 

Contributor
Posts: 41
Registered: ‎02-23-2016

Re: Running Spark in Oozie using yarn-cluster

did you fix this issue?

Contributor
Posts: 41
Registered: ‎02-23-2016

Re: Running Spark in Oozie using yarn-cluster

Dear, I met the same issue, can you explain detailed what you did to fix? Thanks!

Contributor
Posts: 41
Registered: ‎02-23-2016

Re: Running Spark in Oozie using yarn-cluster

Dear sonups3, I met the same issue, can you explain detailed what you did to fix? Thanks!