Support Questions
Find answers, ask questions, and share your expertise

Oozie shell action for sqoop

Contributor

I am trying to run sqoop command by shell script. I am trying to execute that script as shell sction in Oozie. But the job is getting killed. The oozie command is running fine otherwise.

Below is the sqoop command given inside the shell script.

sqoop import --connect jdbc:oracle:thin:@//orcl.*******:1521/orcl --username ****--password ****--table TEST_SQOOP --target-dir /user/hadoop/test -m 1

Below is the job.properties file.

nameNode=hdfs://*******:8020

jobTracker=ip-*******:8032

queueName=default

examplesRoot=examples

oozie.use.system.libpath=true

oozie.libpath=${nameNode}/user/oozie/share/lib

oozie.wf.application.path=${nameNode}/user/${user.name}/generic/oozie_dir

myscript=first.sh

myscriptPath=${oozie.wf.application.path}/first.sh

Note: I am able to run usual operation by shell script without any issues.

5 REPLIES 5

Super Guru

@Gaurab D since you are invoking sqoop from oozie, you must have oozie client installed on all data nodes. Oozie may execute the workflow from any node and that is why you need sqoop client installed on all nodes. Please verify that is complete on your cluster.

Super Guru

@Gaurab D yes but you need to do extra steps:

Shell Action

  • This options requires sqoop client to be installed on all nodes
  • Store Keytabs on HDFS
    • Secure via Ranger/ACL/Chmod
  • Use file tab to identify hdfs keytab location
    • When oozie shell action runs it will download to local yarn directory
    • K-init inside shell script

If you use sqoop to import into hive table then you can do this all in hive action. more on that in this tutorial.

Contributor

@Sunile Manjee I have installed oozie and sqoop client in all the data nodes, but still getting the same error. Is it possible in principle to execute sqoop inside a shell script using oozie!!

Contributor

It is definitely possible. What is the error message you are getting?

Contributor

I am getting below error while running a simple sqoop import command using oracle. That Sqoop command is working fine without oozie.

2016-08-22 08:49:30,013 WARN ShellActionExecutor:523 - SERVER[ip-172-31-31-66.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000121-160802050043050-oozie-oozi-W] ACTION[0000121-160802050043050-oozie-oozi-W@shell-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]