Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

avatar
Explorer

Stack Overflow Question

 

I'm trying to run a simple Sqoop Action through Oozie on Cloudera 5.4.x (Through their QuickStart VM, which should be pre-configured correctly I assume?) When I run the import command via the Sqoop CLI, it all works fine. However, when I attempt to run that same command using an Oozie workflow (through Hue), it fails to find the SqoopMain class.

  

Error log

2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB!
2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import]
2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003]
2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
	... 13 more

 

Workflow action

 

<action name="sqoop-import">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
       <job-tracker>${jobTracker}</job-tracker>
       <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/>
        </prepare>
        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://${oltpHost}/${oltpName}</arg>
        <arg>--username</arg>
        <arg>${oltpUser}</arg>
        <arg>--password</arg>
        <arg>${oltpPassword}</arg>
        <arg>--table</arg>
        <arg>view_et_product_categories</arg>
        <arg>--target-dir</arg>
        <arg>/tmp/etl/${etlUser}/vet_product_categories</arg>
        <arg>--as-avrodatafile</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="done"/>
    <error to="fail"/>
</action>

 

Looked up oozie.service.WorkflowAppService.system.libpath in the cloudera manager, and it was set to /user/oozie - it appends the share/lib to whatever you put in this field, so the full path was /usr/oozie/share/lib The folder in HDFS is versioned with a timestamp. - I'm not sure how oozie adds these classes to the classpath or if it needs additional help to pick this up: /user/oozie/share/lib/lib_20150609033900

1 ACCEPTED SOLUTION

avatar
Explorer

As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true

 

It seems that putting this in the workflow.xml does not work.

 

Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need

View solution in original post

9 REPLIES 9

avatar
Mentor
The share-lib format did change in CDH5. Could you take a look at
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
and let us know if it helps you resolve your issue?

avatar
Explorer
Do I have to include this shared lib path in my Workflow.xml? or should Oozie automatically include it?

Currently, It seems like Oozie does not include this automatically.

avatar
Explorer

As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true

 

It seems that putting this in the workflow.xml does not work.

 

Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need

avatar
Explorer

I am tying to import a test tabl from mysql database. The command was triggered from quickstart vm.

Can someone plase share the exact command or let me know the mistake I am making?

 

Thanks in advance

 

[cloudera@quickstart ~]$ sqoop2 import --connect jdbc:mysql://192.168.112.128/widget --username root --pcloudera --table myhadoop --as-textfile --target-dir /home/cloudera/Documents
Sqoop home directory: /usr/lib/sqoop2
Exception in thread "main" java.io.FileNotFoundException: /home/cloudera/import (Is a directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileReader.<init>(FileReader.java:72)
    at org.apache.sqoop.shell.SqoopShell.interpretFileContent(SqoopShell.java:150)
    at org.apache.sqoop.shell.SqoopShell.main(SqoopShell.java:138)

avatar
Mentor
Please use "sqoop" and not "sqoop2". The latter is a shell, but your
written command is for the 1.x Sqoop features.

avatar
Explorer
sqoop is not working correctly and seems due to missing env. variable as
below:
any insight?

[cloudera@quickstart Documents]$ sqoop import --connect jdbc:mysql://
192.168.112.128/widget --username root --pcloudera --table myhadoop
--as-textfile --target-dir /home/cloudera/Documents
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will
fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/02/15 21:10:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Error parsing arguments for
import:
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--pcloudera
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: --table
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: myhadoop
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--as-textfile
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--target-dir
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
/home/cloudera/Documents

avatar
Explorer
Also. how to find out hostname for mysql from command line in my quickstart vm.
In the above command 192.168.112.128 is the output of ifconfig -a which I am not fully convinced is the right one?

thanks

avatar
Explorer

Any assistnce t find out

jdbc url for mysql connectivity through sqoop in quickstart vm?

 

thanks

avatar
New Contributor

Hi,

 

You must place the connector jar inside the shared folder for oozie which is hdfs:

 

hdfs://user/oozie/sqoop/lib/lib<instance>/