Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie Sqoop action, using home brewed Sqoop plugin

Highlighted

Oozie Sqoop action, using home brewed Sqoop plugin

New Contributor

Hello

I have a custom Sqoop plugin, I want to use in an Oozie Sqoop action. Whenever i run the workflow, i get the following error message in the Oozie jobhistory log:

=================================================================

>>> Invoking Sqoop command line now >>>

1828 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2016-08-16 10:35:31,504 WARN  [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

I have tried running the same workflow, but changing the name of the first argument (toolname), to something random (ie. running "sqoop asdfsdf --target-dir /tmp/"), and the error is the same. Thus it seems to me, that it fails at reading the .jar file, which contains the plugin.

As i was unsure where to place the .jar plugin file, i placed it in several directories: 1) hdfs: /user/oozie/share/lib/[library_folder] 2) namenode: /usr/hdp/2.3.0.0-2557/oozie/share/lib/sqoop, /usr/hdp/2.3.0.0-2557/oozie/lib/ I created the tools.d folder: /usr/hdp/2.3.0.0-2557/oozie/share/conf/tools.d, containing an xml file, containing one line: "[package name]=[location of .jar file]"

From the same logs i can see the .jar file appear in the list of files:

Files in current dir:/grid/XXX/XXX/XXX/XXX
======================
File: jersey-core-1.9.jar
File: libthrift-0.9.2.jar
....
File: MyPlugin.jar
....

I have also looked in the log files of yarn ("oozie:launcher"), which gives me no further info.

Do you have any advice on how to make this work?

Don't have an account?
Coming from Hortonworks? Activate your account here