Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How to run spark action through oozie..?

Explorer

Hi All,

I am new to oozie trying to run my simple word count spark job through oozie ,i am successfully submitting the job, but i am getting this exception Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found,

Here is my workflow.xml and job.properties i followed these link to run the spark job

https://community.hortonworks.com/articles/48920/how-to-run-spark-action-in-oozie-of-hdp-230.html

we tried a sample hive job through oozie we successfully executed that, for this i am getting this error.please let me know the root cause of this exception, for this i have tried different test cases,we copied spark-assembly-jar to usr/oozie/share/lib although i am getting this exception,error.txt please find attached error log aso

Workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.5' name='Sparkjob'>
    <start to='spark-node' />
    <action name='spark-node'>
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local[*]</master>
<mode>client</mode>
            <name>Spark-FileCopy</name>
            <class>org.examples.WordCounte</class>
            <jar>${nameNode}/anji/oozie/lib/Spark_test.jar</jar>
            <arg>${input}</arg>
            <arg>${output}</arg>
        </spark>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Workflow failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]
        </message>
    </kill>
    <end name='end' />
</workflow-app>

job.properties:nameNode=hdfs://nn:8020
jobTracker=jT:8032
queueName=default
input =${nameNode}/sample2.txt
output=${nameNode}/output3
oozie.system.lib.path = true
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.action.sharelib.for.spark=${namenode}/user/oozie/share/lib/spark/
dryrun=False
oozie.wf.application.path=hdfs://quickstart.cloudera:8020/anji/oozie/








5 REPLIES 5

Super Guru
@AnjiReddy Anumolu

I think you need the following (you are missing "use").

oozie.use.system.lib.path =true

Explorer

Hi Quereshi,

it's a just typo error , i am using that "use" also

Super Guru

@AnjiReddy Anumolu

In that case, what's the value of

oozie.service.WorkflowAppService.system.libpath?

https://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml

Explorer

Please follow below steps, it should resolve the issue.

1) Create a "lib" folder directly next to workflow.xml

2) Copy sparkUber.jar to lib folder

3) In the job.properties file, add oozie.use.system.libpath=true

4) hdfs dfs -put /path/to/sparkUber.jar /<oozie_worflow_name>/lib/

Explorer

Hi Amit,

what is that sparkUber.jar i copied spark-assembly jar to /oozie/share/lib/spark folder

please let me know about that sparkUber.jar