Reply
Highlighted
Contributor
Posts: 41
Registered: ‎02-23-2016

Class org.apache.oozie.action.hadoop.SparkMain not found when using CDH 5.5.0 Hue and oozie

Dear Cloudera,

 

I set following values in job.properties and in action properties in CDH 5.5.0 Hue oozie:

oozie.use.system.libpath=true

oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935

oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark

 

Here is the workflow file:

<workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5">

    <start to="spark-b23b"/>

    <kill name="Kill">

        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <action name="spark-b23b">

        <spark xmlns="uri:oozie:spark-action:0.1">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <configuration>

                <property>

                    <name>spark.executor.extraClassPath</name>

                    <value>lib/guava-16.0.1.jar</value>

                </property>

                <property>

                    <name>oozie.action.sharelib.for.spark</name>

                    <value>/user/oozie/share/lib/lib_20151201085935/spark</value>

                </property>

                <property>

                    <name>oozie.use.system.libpath</name>

                    <value>true</value>

                </property>

                <property>

                    <name>oozie.libpath</name>

                    <value>/user/oozie/share/lib/lib_20151201085935</value>

                </property>

            </configuration>

            <master>local[4]</master>

            <mode>client</mode>

            <name>sparktest-cassandra</name>

              <class>TestCassandra</class>

            <jar>lib/sparktest.jar</jar>

              <spark-opts>--driver-class-path /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar</spark-opts>

              <arg>s3n://gridx-output/sparktest/ </arg>

              <arg>10</arg>

              <arg>3</arg>

              <arg>2</arg>

        </spark>

        <ok to="End"/>

        <error to="Kill"/>

    </action>

    <end name="End"/>

</workflow-app>

 

job.properties file:

oozie.use.system.libpath=true

security_enabled=False

oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935

oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark

dryrun=False

jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032

nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020

 

 

But I got following SparkMain class not found issue:

 

<<< Invocation of Main class completed <<<

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)

at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)

at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)

... 13 more

 

Oozie Launcher failed, finishing Hadoop job gracefully

 

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/admin/oozie-oozi/0000006-160224085347053-o...

 

Oozie Launcher ends

 

 

 

Can you help to give any suggestion? Thanks a lot!

Contributor
Posts: 41
Registered: ‎02-23-2016

Re: Class org.apache.oozie.action.hadoop.SparkMain not found when using CDH 5.5.0 Hue and oozie

need help in CDH Hue workflow...