Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to run spark action through oozie..?

How to run spark action through oozie..?

New Contributor

Hi All,

I am new to oozie trying to run my simple word count spark job through oozie ,i am successfully submitting the job, but i am getting this exception Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found,

Here is my workflow.xml and job.properties i followed these link to run the spark job

https://community.hortonworks.com/articles/48920/how-to-run-spark-action-in-oozie-of-hdp-230.html

we tried a sample hive job through oozie we successfully executed that, for this i am getting this error.please let me know the root cause of this exception, for this i have tried different test cases,we copied spark-assembly-jar to usr/oozie/share/lib although i am getting this exception,error.txt please find attached error log aso

Workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.5' name='Sparkjob'>
    <start to='spark-node' />
    <action name='spark-node'>
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local[*]</master>
<mode>client</mode>
            <name>Spark-FileCopy</name>
            <class>org.examples.WordCounte</class>
            <jar>${nameNode}/anji/oozie/lib/Spark_test.jar</jar>
            <arg>${input}</arg>
            <arg>${output}</arg>
        </spark>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Workflow failed, error
            message[${wf:errorMessage(wf:lastErrorNode())}]
        </message>
    </kill>
    <end name='end' />
</workflow-app>

job.properties:nameNode=hdfs://nn:8020
jobTracker=jT:8032
queueName=default
input =${nameNode}/sample2.txt
output=${nameNode}/output3
oozie.system.lib.path = true
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.action.sharelib.for.spark=${namenode}/user/oozie/share/lib/spark/
dryrun=False
oozie.wf.application.path=hdfs://quickstart.cloudera:8020/anji/oozie/








5 REPLIES 5

Re: How to run spark action through oozie..?

Super Guru
@AnjiReddy Anumolu

I think you need the following (you are missing "use").

oozie.use.system.lib.path =true

Re: How to run spark action through oozie..?

New Contributor

Hi Quereshi,

it's a just typo error , i am using that "use" also

Highlighted

Re: How to run spark action through oozie..?

Super Guru

@AnjiReddy Anumolu

In that case, what's the value of

oozie.service.WorkflowAppService.system.libpath?

https://oozie.apache.org/docs/3.2.0-incubating/oozie-default.xml

Re: How to run spark action through oozie..?

New Contributor

Please follow below steps, it should resolve the issue.

1) Create a "lib" folder directly next to workflow.xml

2) Copy sparkUber.jar to lib folder

3) In the job.properties file, add oozie.use.system.libpath=true

4) hdfs dfs -put /path/to/sparkUber.jar /<oozie_worflow_name>/lib/

Re: How to run spark action through oozie..?

New Contributor

Hi Amit,

what is that sparkUber.jar i copied spark-assembly jar to /oozie/share/lib/spark folder

please let me know about that sparkUber.jar