Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

SOLVED Go to solution

CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

New Contributor

Stack Overflow Question

 

I'm trying to run a simple Sqoop Action through Oozie on Cloudera 5.4.x (Through their QuickStart VM, which should be pre-configured correctly I assume?) When I run the import command via the Sqoop CLI, it all works fine. However, when I attempt to run that same command using an Oozie workflow (through Hue), it fails to find the SqoopMain class.

  

Error log

2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB!
2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import]
2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003]
2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
	... 13 more

 

Workflow action

 

<action name="sqoop-import">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
       <job-tracker>${jobTracker}</job-tracker>
       <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/>
        </prepare>
        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://${oltpHost}/${oltpName}</arg>
        <arg>--username</arg>
        <arg>${oltpUser}</arg>
        <arg>--password</arg>
        <arg>${oltpPassword}</arg>
        <arg>--table</arg>
        <arg>view_et_product_categories</arg>
        <arg>--target-dir</arg>
        <arg>/tmp/etl/${etlUser}/vet_product_categories</arg>
        <arg>--as-avrodatafile</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="done"/>
    <error to="fail"/>
</action>

 

Looked up oozie.service.WorkflowAppService.system.libpath in the cloudera manager, and it was set to /user/oozie - it appends the share/lib to whatever you put in this field, so the full path was /usr/oozie/share/lib The folder in HDFS is versioned with a timestamp. - I'm not sure how oozie adds these classes to the classpath or if it needs additional help to pick this up: /user/oozie/share/lib/lib_20150609033900

1 ACCEPTED SOLUTION

Accepted Solutions

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

New Contributor

As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true

 

It seems that putting this in the workflow.xml does not work.

 

Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need

9 REPLIES 9

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Master Guru
The share-lib format did change in CDH5. Could you take a look at
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
and let us know if it helps you resolve your issue?

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

New Contributor
Do I have to include this shared lib path in my Workflow.xml? or should Oozie automatically include it?

Currently, It seems like Oozie does not include this automatically.

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

New Contributor

As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true

 

It seems that putting this in the workflow.xml does not work.

 

Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Explorer

I am tying to import a test tabl from mysql database. The command was triggered from quickstart vm.

Can someone plase share the exact command or let me know the mistake I am making?

 

Thanks in advance

 

[cloudera@quickstart ~]$ sqoop2 import --connect jdbc:mysql://192.168.112.128/widget --username root --pcloudera --table myhadoop --as-textfile --target-dir /home/cloudera/Documents
Sqoop home directory: /usr/lib/sqoop2
Exception in thread "main" java.io.FileNotFoundException: /home/cloudera/import (Is a directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileReader.<init>(FileReader.java:72)
    at org.apache.sqoop.shell.SqoopShell.interpretFileContent(SqoopShell.java:150)
    at org.apache.sqoop.shell.SqoopShell.main(SqoopShell.java:138)

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Master Guru
Please use "sqoop" and not "sqoop2". The latter is a shell, but your
written command is for the 1.x Sqoop features.

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Explorer
sqoop is not working correctly and seems due to missing env. variable as
below:
any insight?

[cloudera@quickstart Documents]$ sqoop import --connect jdbc:mysql://
192.168.112.128/widget --username root --pcloudera --table myhadoop
--as-textfile --target-dir /home/cloudera/Documents
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will
fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/02/15 21:10:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Error parsing arguments for
import:
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--pcloudera
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: --table
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: myhadoop
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--as-textfile
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--target-dir
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
/home/cloudera/Documents
Highlighted

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Explorer
Also. how to find out hostname for mysql from command line in my quickstart vm.
In the above command 192.168.112.128 is the output of ifconfig -a which I am not fully convinced is the right one?

thanks

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

Explorer

Any assistnce t find out

jdbc url for mysql connectivity through sqoop in quickstart vm?

 

thanks

Re: CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain

New Contributor

Hi,

 

You must place the connector jar inside the shared folder for oozie which is hdfs:

 

hdfs://user/oozie/sqoop/lib/lib<instance>/