CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

[#1]

Stack Overflow Question

I'm trying to run a simple Sqoop Action through Oozie on Cloudera 5.4.x (Through their QuickStart VM, which should be pre-configured correctly I assume?) When I run the import command via the Sqoop CLI, it all works fine. However, when I attempt to run that same command using an Oozie workflow (through Hue), it fails to find the SqoopMain class.

Error log

2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB!
2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import]
2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003]
2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
	... 13 more

Workflow action

<action name="sqoop-import">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
       <job-tracker>${jobTracker}</job-tracker>
       <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/>
        </prepare>
        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://${oltpHost}/${oltpName}</arg>
        <arg>--username</arg>
        <arg>${oltpUser}</arg>
        <arg>--password</arg>
        <arg>${oltpPassword}</arg>
        <arg>--table</arg>
        <arg>view_et_product_categories</arg>
        <arg>--target-dir</arg>
        <arg>/tmp/etl/${etlUser}/vet_product_categories</arg>
        <arg>--as-avrodatafile</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="done"/>
    <error to="fail"/>
</action>

Looked up oozie.service.WorkflowAppService.system.libpath in the cloudera manager, and it was set to /user/oozie - it appends the share/lib to whatever you put in this field, so the full path was /usr/oozie/share/lib The folder in HDFS is versioned with a timestamp. - I'm not sure how oozie adds these classes to the classpath or if it needs additional help to pick this up: /user/oozie/share/lib/lib_20150609033900

[#2]

The share-lib format did change in CDH5. Could you take a look at
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
and let us know if it helps you resolve your issue?

[#3]

Do I have to include this shared lib path in my Workflow.xml? or should Oozie automatically include it?

Currently, It seems like Oozie does not include this automatically.

[#4 · Accepted solution]

As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true

It seems that putting this in the workflow.xml does not work.

Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need

[#5]

I am tying to import a test tabl from mysql database. The command was triggered from quickstart vm.

Can someone plase share the exact command or let me know the mistake I am making?

Thanks in advance

[cloudera@quickstart ~]$ sqoop2 import --connect jdbc:mysql://192.168.112.128/widget --username root --pcloudera --table myhadoop --as-textfile --target-dir /home/cloudera/Documents
Sqoop home directory: /usr/lib/sqoop2
Exception in thread "main" java.io.FileNotFoundException: /home/cloudera/import (Is a directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileReader.<init>(FileReader.java:72)
    at org.apache.sqoop.shell.SqoopShell.interpretFileContent(SqoopShell.java:150)
    at org.apache.sqoop.shell.SqoopShell.main(SqoopShell.java:138)

[#6]

Please use "sqoop" and not "sqoop2". The latter is a shell, but your
written command is for the 1.x Sqoop features.

[#7]

sqoop is not working correctly and seems due to missing env. variable as
below:
any insight?

[cloudera@quickstart Documents]$ sqoop import --connect jdbc:mysql://
192.168.112.128/widget --username root --pcloudera --table myhadoop
--as-textfile --target-dir /home/cloudera/Documents
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will
fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/02/15 21:10:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Error parsing arguments for
import:
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--pcloudera
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: --table
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: myhadoop
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--as-textfile
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--target-dir
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
/home/cloudera/Documents

[#8]

Also. how to find out hostname for mysql from command line in my quickstart vm.
In the above command 192.168.112.128 is the output of ifconfig -a which I am not fully convinced is the right one?

thanks

[#9]

Any assistnce t find out

jdbc url for mysql connectivity through sqoop in quickstart vm?

thanks

[#10]

Hi,

You must place the connector jar inside the shared folder for oozie which is hdfs:

hdfs://user/oozie/sqoop/lib/lib<instance>/