Reply
Contributor
Posts: 26
Registered: ‎05-11-2016

Not able to run Spark jar action in Oozie

I am trying to run the Spark action in Oozie with spark .jar

 

Provided below values in Hue Oozie Spark action,

Jar/py Name: solution.jar

Main Class: Module.final_solution

Files: /user/hadoop/solution/solution.jar

 

Below are the properties:

Spark Master: yarn-master

Mode: Cluster

App Name: Final Solution

 

Getting below error while running Oozie action with above configurations,

error.jpg

 

Same jar working fine from Edge node with Spark submit.

Any help appriciated.

New Contributor
Posts: 5
Registered: ‎08-24-2017

Re: Not able to run Spark jar action in Oozie

This looks like Hue might be having problems communicating with Yarn. Have you tried submitting the job.properties from the Edge node using

oozie job -submit

?

 

Is hue able to submit a hello-world oozie job and monitor it? Are you able to submit other jobs via Hue (Hive/pig, etc)?

Contributor
Posts: 26
Registered: ‎05-11-2016

Re: Not able to run Spark jar action in Oozie

Yes, other jobs are wokring fine.

I am on CDH 5.9, so just tried with jar path in options list which is atleast running the job now.

I am encountering below error while running the job,

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, requirement failed
java.lang.IllegalArgumentException: requirement failed

 

 

Highlighted
Cloudera Employee
Posts: 22
Registered: ‎06-17-2016

Re: Not able to run Spark jar action in Oozie

Can you post a full stack trace?

Announcements