Support Questions
Find answers, ask questions, and share your expertise

Oozie Sqoop action, using home brewed Sqoop plugin

Oozie Sqoop action, using home brewed Sqoop plugin

New Contributor

Hello

I have a custom Sqoop plugin, I want to use in an Oozie Sqoop action. Whenever i run the workflow, i get the following error message in the Oozie jobhistory log:

=================================================================

>>> Invoking Sqoop command line now >>>

1828 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2016-08-16 10:35:31,504 WARN  [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

I have tried running the same workflow, but changing the name of the first argument (toolname), to something random (ie. running "sqoop asdfsdf --target-dir /tmp/"), and the error is the same. Thus it seems to me, that it fails at reading the .jar file, which contains the plugin.

As i was unsure where to place the .jar plugin file, i placed it in several directories: 1) hdfs: /user/oozie/share/lib/[library_folder] 2) namenode: /usr/hdp/2.3.0.0-2557/oozie/share/lib/sqoop, /usr/hdp/2.3.0.0-2557/oozie/lib/ I created the tools.d folder: /usr/hdp/2.3.0.0-2557/oozie/share/conf/tools.d, containing an xml file, containing one line: "[package name]=[location of .jar file]"

From the same logs i can see the .jar file appear in the list of files:

Files in current dir:/grid/XXX/XXX/XXX/XXX
======================
File: jersey-core-1.9.jar
File: libthrift-0.9.2.jar
....
File: MyPlugin.jar
....

I have also looked in the log files of yarn ("oozie:launcher"), which gives me no further info.

Do you have any advice on how to make this work?