Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to run spark action through oozie..?

avatar
Rising Star

Hi All,

I am new to oozie trying to run my simple word count spark job through oozie ,i am successfully submitting the job, but i am getting this exception Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found,

Here is my workflow.xml and job.properties i followed these link to run the spark job

https://community.hortonworks.com/articles/48920/how-to-run-spark-action-in-oozie-of-hdp-230.html

we tried a sample hive job through oozie we successfully executed that, for this i am getting this error.please let me know the root cause of this exception, for this i have tried different test cases,we copied spark-assembly-jar to usr/oozie/share/lib although i am getting this exception,error.txt please find attached error log aso

Workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.5' name='Sparkjob'>
    <start to='spark-node' />
    <action name='spark-node'>
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local[*]</master>
<mode>client</mode>
            <name>Spark-FileCopy</name>
            <class>org.examples.WordCounte</class>
            <jar>${nameNode}/anji/oozie/lib/Spark_test.jar</jar>
            <arg>${input}</arg>
            <arg>${output}</arg>
        </spark>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Workflow failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]
        </message>
    </kill>
    <end name='end' />
</workflow-app>

job.properties:nameNode=hdfs://nn:8020
jobTracker=jT:8032
queueName=default
input =${nameNode}/sample2.txt
output=${nameNode}/output3
oozie.system.lib.path = true
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.action.sharelib.for.spark=${namenode}/user/oozie/share/lib/spark/
dryrun=False
oozie.wf.application.path=hdfs://quickstart.cloudera:8020/anji/oozie/








5 REPLIES 5

avatar
Super Guru
@AnjiReddy Anumolu

I think you need the following (you are missing "use").

oozie.use.system.lib.path =true

avatar
Rising Star

Hi Quereshi,

it's a just typo error , i am using that "use" also

avatar
Super Guru

@AnjiReddy Anumolu

In that case, what's the value of

oozie.service.WorkflowAppService.system.libpath?

https://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml

avatar
Contributor

Please follow below steps, it should resolve the issue.

1) Create a "lib" folder directly next to workflow.xml

2) Copy sparkUber.jar to lib folder

3) In the job.properties file, add oozie.use.system.libpath=true

4) hdfs dfs -put /path/to/sparkUber.jar /<oozie_worflow_name>/lib/

avatar
Rising Star

Hi Amit,

what is that sparkUber.jar i copied spark-assembly jar to /oozie/share/lib/spark folder

please let me know about that sparkUber.jar