- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
CDH 5.4.4 Oozie unable to run Scoop action - ClassNotFound SqoopMain
- Labels:
-
Apache Sqoop
Created on 07-16-2015 02:44 PM - edited 09-16-2022 02:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm trying to run a simple Sqoop Action through Oozie on Cloudera 5.4.x (Through their QuickStart VM, which should be pre-configured correctly I assume?) When I run the import command via the Sqoop CLI, it all works fine. However, when I attempt to run that same command using an Oozie workflow (through Hue), it fails to find the SqoopMain class.
Error log
2015-07-14 14:58:02,997 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] [***0000001-150714084022371-oozie-oozi-W@sqoop-import***]Action updated in DB! 2015-07-14 14:58:12,802 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] callback for action [0000001-150714084022371-oozie-oozi-W@sqoop-import] 2015-07-14 14:58:13,058 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] action completed, external ID [job_1436888351169_0003] 2015-07-14 14:58:13,078 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found 2015-07-14 14:58:13,085 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: SERVER[quickstart.cloudera] USER[cloudera] GROUP[-] TOKEN[] APP[simpleWF] JOB[0000001-150714084022371-oozie-oozi-W] ACTION[0000001-150714084022371-oozie-oozi-W@sqoop-import] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181) at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) ... 13 more
Workflow action
<action name="sqoop-import"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <prepare> <delete path="${nameNode}/tmp/etl/${etlUser}/vet_product_categories"/> </prepare> <arg>import</arg> <arg>--connect</arg> <arg>jdbc:mysql://${oltpHost}/${oltpName}</arg> <arg>--username</arg> <arg>${oltpUser}</arg> <arg>--password</arg> <arg>${oltpPassword}</arg> <arg>--table</arg> <arg>view_et_product_categories</arg> <arg>--target-dir</arg> <arg>/tmp/etl/${etlUser}/vet_product_categories</arg> <arg>--as-avrodatafile</arg> <arg>-m</arg> <arg>1</arg> </sqoop> <ok to="done"/> <error to="fail"/> </action>
Looked up oozie.service.WorkflowAppService.system.libpath in the cloudera manager, and it was set to /user/oozie - it appends the share/lib to whatever you put in this field, so the full path was /usr/oozie/share/lib The folder in HDFS is versioned with a timestamp. - I'm not sure how oozie adds these classes to the classpath or if it needs additional help to pick this up: /user/oozie/share/lib/lib_20150609033900
Created 07-27-2015 11:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true
It seems that putting this in the workflow.xml does not work.
Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need
Created 07-16-2015 08:54 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
and let us know if it helps you resolve your issue?
Created 07-27-2015 07:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Currently, It seems like Oozie does not include this automatically.
Created 07-27-2015 11:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As it turns out, you need to supply a job.properties with oozie.use.system.libpath=true
It seems that putting this in the workflow.xml does not work.
Basically, what I have now is a bare sqoop action with no configuration and a job.properties with all properties i need
Created 02-15-2016 08:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am tying to import a test tabl from mysql database. The command was triggered from quickstart vm.
Can someone plase share the exact command or let me know the mistake I am making?
Thanks in advance
[cloudera@quickstart ~]$ sqoop2 import --connect jdbc:mysql://192.168.112.128/widget --username root --pcloudera --table myhadoop --as-textfile --target-dir /home/cloudera/Documents
Sqoop home directory: /usr/lib/sqoop2
Exception in thread "main" java.io.FileNotFoundException: /home/cloudera/import (Is a directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileReader.<init>(FileReader.java:72)
at org.apache.sqoop.shell.SqoopShell.interpretFileContent(SqoopShell.java:150)
at org.apache.sqoop.shell.SqoopShell.main(SqoopShell.java:138)
Created 02-15-2016 09:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
written command is for the 1.x Sqoop features.
Created 02-15-2016 09:13 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
below:
any insight?
[cloudera@quickstart Documents]$ sqoop import --connect jdbc:mysql://
192.168.112.128/widget --username root --pcloudera --table myhadoop
--as-textfile --target-dir /home/cloudera/Documents
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will
fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/02/15 21:10:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Error parsing arguments for
import:
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--pcloudera
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: --table
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument: myhadoop
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--as-textfile
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
--target-dir
16/02/15 21:10:31 ERROR tool.BaseSqoopTool: Unrecognized argument:
/home/cloudera/Documents
Created 02-15-2016 09:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the above command 192.168.112.128 is the output of ifconfig -a which I am not fully convinced is the right one?
thanks
Created 02-17-2016 11:05 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any assistnce t find out
jdbc url for mysql connectivity through sqoop in quickstart vm?
thanks
Created 08-10-2017 03:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
You must place the connector jar inside the shared folder for oozie which is hdfs:
hdfs://user/oozie/sqoop/lib/lib<instance>/