Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Possible to call sqoop2 with passing batch script in Oozie?

Possible to call sqoop2 with passing batch script in Oozie?

New Contributor

I've been trying to get a shell action configured properly to initiate a sqoop2 job.  I've been able to run it just fine on the edge node, but when I try a similar approach in the action configuration, I get nothing but errors either about the file not existing (even though it does), or worse, it tries to parse my sqoop2 commands as actual linux commands (example: start job --j ## is attempted to be run as start for Launchpad...).

 

The configuration of the shell action is simply /usr/bin/sqoop2 client ./Sqoop2Commands.sqoop.

I've attached the Sqoop2Commands.sqoop file to the files section for the action.

 

When I execute the workflow, in this instance, it comes back with the following:

 

start: invalid option: --j
Try `start --help' for more information.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

It's basically trying to parse all of my sqoop2 commands in the sqoop file as actual linux commands.

 

Has anyone been able to successfully start a sqoop2 job from the shell action?

 

6 REPLIES 6

Re: Possible to call sqoop2 with passing batch script in Oozie?

Expert Contributor

I don't seem to see your sqoop2commands.sqoop file. Could you copy/paste the contents here?

Re: Possible to call sqoop2 with passing batch script in Oozie?

New Contributor

It's extremely basic...

 

set server --host masked_host_name_here --port 12000 --webapp sqoop
start job --j 21

This same sqoop file works just fine when running on the edge node with sqoop2 ./sqoop2commands.sqoop

 

As mentioned before, when attempting the same with a shell action within Oozie, it tries to parse each of those lines as an actual command, not a sqoop2 statement.

Re: Possible to call sqoop2 with passing batch script in Oozie?

Expert Contributor

Could you try replacing '--j' with '-j'?

Re: Possible to call sqoop2 with passing batch script in Oozie?

New Contributor

Here's stderr from Oozie:

 

start: invalid option: -j
Try `start --help' for more information.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

 

updated sqoop file:

 

set server --host masked_host_name --port 12000 --webapp sqoop
start job -j 21

 

 

Re: Possible to call sqoop2 with passing batch script in Oozie?

Expert Contributor
So I just tried this out for myself and it seems to work fine (in a pseudo distributed cluster). What version of CDH are you using?

Re: Possible to call sqoop2 with passing batch script in Oozie?

New Contributor

I believe 4.6.3?