Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Class org.apache.oozie.action.hadoop.SparkMain not found when using CDH 5.5.0 Hue and oozie

Class org.apache.oozie.action.hadoop.SparkMain not found when using CDH 5.5.0 Hue and oozie

Explorer

Dear Cloudera,

 

I set following values in job.properties and in action properties in CDH 5.5.0 Hue oozie:

oozie.use.system.libpath=true

oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935

oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark

 

Here is the workflow file:

<workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5">

    <start to="spark-b23b"/>

    <kill name="Kill">

        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <action name="spark-b23b">

        <spark xmlns="uri:oozie:spark-action:0.1">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <configuration>

                <property>

                    <name>spark.executor.extraClassPath</name>

                    <value>lib/guava-16.0.1.jar</value>

                </property>

                <property>

                    <name>oozie.action.sharelib.for.spark</name>

                    <value>/user/oozie/share/lib/lib_20151201085935/spark</value>

                </property>

                <property>

                    <name>oozie.use.system.libpath</name>

                    <value>true</value>

                </property>

                <property>

                    <name>oozie.libpath</name>

                    <value>/user/oozie/share/lib/lib_20151201085935</value>

                </property>

            </configuration>

            <master>local[4]</master>

            <mode>client</mode>

            <name>sparktest-cassandra</name>

              <class>TestCassandra</class>

            <jar>lib/sparktest.jar</jar>

              <spark-opts>--driver-class-path /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar</spark-opts>

              <arg>s3n://gridx-output/sparktest/ </arg>

              <arg>10</arg>

              <arg>3</arg>

              <arg>2</arg>

        </spark>

        <ok to="End"/>

        <error to="Kill"/>

    </action>

    <end name="End"/>

</workflow-app>

 

job.properties file:

oozie.use.system.libpath=true

security_enabled=False

oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935

oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark

dryrun=False

jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032

nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020

 

 

But I got following SparkMain class not found issue:

 

<<< Invocation of Main class completed <<<

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)

at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)

... 13 more

 

Oozie Launcher failed, finishing Hadoop job gracefully

 

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/admin/oozie-oozi/0000006-160224085347053-o...

 

Oozie Launcher ends

 

 

 

Can you help to give any suggestion? Thanks a lot!

1 REPLY 1

Re: Class org.apache.oozie.action.hadoop.SparkMain not found when using CDH 5.5.0 Hue and oozie

Explorer

need help in CDH Hue workflow...

Don't have an account?
Coming from Hortonworks? Activate your account here