Reply
New Contributor
Posts: 6
Registered: ‎05-21-2014

Possible to call sqoop2 with passing batch script in Oozie?

I've been trying to get a shell action configured properly to initiate a sqoop2 job.  I've been able to run it just fine on the edge node, but when I try a similar approach in the action configuration, I get nothing but errors either about the file not existing (even though it does), or worse, it tries to parse my sqoop2 commands as actual linux commands (example: start job --j ## is attempted to be run as start for Launchpad...).

 

The configuration of the shell action is simply /usr/bin/sqoop2 client ./Sqoop2Commands.sqoop.

I've attached the Sqoop2Commands.sqoop file to the files section for the action.

 

When I execute the workflow, in this instance, it comes back with the following:

 

start: invalid option: --j
Try `start --help' for more information.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

It's basically trying to parse all of my sqoop2 commands in the sqoop file as actual linux commands.

 

Has anyone been able to successfully start a sqoop2 job from the shell action?

 

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: Possible to call sqoop2 with passing batch script in Oozie?

I don't seem to see your sqoop2commands.sqoop file. Could you copy/paste the contents here?

New Contributor
Posts: 6
Registered: ‎05-21-2014

Re: Possible to call sqoop2 with passing batch script in Oozie?

It's extremely basic...

 

set server --host masked_host_name_here --port 12000 --webapp sqoop
start job --j 21

This same sqoop file works just fine when running on the edge node with sqoop2 ./sqoop2commands.sqoop

 

As mentioned before, when attempting the same with a shell action within Oozie, it tries to parse each of those lines as an actual command, not a sqoop2 statement.

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: Possible to call sqoop2 with passing batch script in Oozie?

Could you try replacing '--j' with '-j'?

New Contributor
Posts: 6
Registered: ‎05-21-2014

Re: Possible to call sqoop2 with passing batch script in Oozie?

Here's stderr from Oozie:

 

start: invalid option: -j
Try `start --help' for more information.
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

 

updated sqoop file:

 

set server --host masked_host_name --port 12000 --webapp sqoop
start job -j 21

 

 

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: Possible to call sqoop2 with passing batch script in Oozie?

So I just tried this out for myself and it seems to work fine (in a pseudo distributed cluster). What version of CDH are you using?
New Contributor
Posts: 6
Registered: ‎05-21-2014

Re: Possible to call sqoop2 with passing batch script in Oozie?

I believe 4.6.3?

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.