Reply
New Contributor
Posts: 2
Registered: ‎07-16-2016

Sqoop job schedule steps through ozzie

[ Edited ]

Hi All,

 

I am new to hadoop i just wanted to learn how to schedule sqoop job in ozzie which takes data from rdbms like mysql and then put inot hive table.

Just wanted to know step by step procedure to learn.

 

just for practice i went to workflow and created a workflow with below parameter 

 

a) put the my sql jar under hdfs /user/oozie/share/lib/sqoop

 

b) created the workflow with below commands

command :  sqoop import --connect jdbc:mysql://localhost/training --username user --password  *** --table table1 --hive-import --hive-overwrite --hive-table training.table1 -m 1 

 

prepare :left blank

params :left blank

job properties :left blank

files  :left blank

Archives :leftblank

job xml left : blank 

 

c) submitted the workflow 

<workflow-app name="TestSqoop" xmlns="uri:oozie:workflow:0.4">
<start to="TestSqoop"/>
<action name="TestSqoop">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>sqoop import --connect jdbc:mysql://localhost/training --username user --password ******** --table table1 --hive-import --hive-overwrite --hive-table training.table1 -m 1
</command>
<archive>#</archive>
</sqoop>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

 

configuration

 

Name Value
hue-id-w 4
jobTracker localhost.localdomain:8021
nameNode hdfs://localhost.localdomain:8020
oozie.use.system.libpath true
oozie.wf.application.path hdfs://localhost.localdomain:8020/user/hue/oozie/workspaces/_cloudera_-oozie-4-1468679262.27
user.name cloudera

 

d) it went to error with below logs 

 

stderr

 

No such sqoop tool: sqoop. See 'sqoop help'.
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

 

stdout 

4370 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

 

logs 

2016-07-16 09:15:30,717 INFO org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@:start:] Start action [0000014-160716070646797-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-07-16 09:15:30,718 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@:start:] [***0000014-160716070646797-oozie-oozi-W@:start:***]Action status=DONE
2016-07-16 09:15:30,718 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@:start:] [***0000014-160716070646797-oozie-oozi-W@:start:***]Action updated in DB!
2016-07-16 09:15:31,084 INFO org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] Start action [0000014-160716070646797-oozie-oozi-W@TestSqoop] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-07-16 09:15:34,034 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] credentials is null for the action
2016-07-16 09:15:36,235 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] checking action, external ID [job_201607160700_0025] status [RUNNING]
2016-07-16 09:15:36,339 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] [***0000014-160716070646797-oozie-oozi-W@TestSqoop***]Action status=RUNNING
2016-07-16 09:15:36,339 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] [***0000014-160716070646797-oozie-oozi-W@TestSqoop***]Action updated in DB!
2016-07-16 09:15:58,290 INFO org.apache.oozie.servlet.CallbackServlet: USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] callback for action [0000014-160716070646797-oozie-oozi-W@TestSqoop]
2016-07-16 09:15:58,392 INFO org.apache.oozie.action.hadoop.SqoopActionExecutor: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] action completed, external ID [job_201607160700_0025]
2016-07-16 09:15:58,427 WARN org.apache.oozie.action.hadoop.SqoopActionExecutor: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2016-07-16 09:15:58,589 INFO org.apache.oozie.command.wf.ActionEndXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@TestSqoop] ERROR is considered as FAILED for SLA
2016-07-16 09:15:58,651 INFO org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@kill] Start action [0000014-160716070646797-oozie-oozi-W@kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-07-16 09:15:58,651 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@kill] [***0000014-160716070646797-oozie-oozi-W@kill***]Action status=DONE
2016-07-16 09:15:58,651 WARN org.apache.oozie.command.wf.ActionStartXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[0000014-160716070646797-oozie-oozi-W@kill] [***0000014-160716070646797-oozie-oozi-W@kill***]Action updated in DB!
2016-07-16 09:15:58,731 WARN org.apache.oozie.command.coord.CoordActionUpdateXCommand: USER[cloudera] GROUP[-] TOKEN[] APP[TestSqoop] JOB[0000014-160716070646797-oozie-oozi-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100

 

 

I know i am missing loft of things here please help me out with this 

 

 

Explorer
Posts: 12
Registered: ‎09-02-2016

Re: Sqoop job schedule steps through ozzie

Hi hdr1,

 

In your sqoop command please remove sqoop and try.

 

command :  import --connect jdbc:mysql://localhost/training --username user --password  *** --table table1 --hive-import --hive-overwrite --hive-table training.table1 -m 1 

New Contributor
Posts: 1
Registered: ‎01-04-2019

Re: Sqoop job schedule steps through ozzie

I also having issue, I've removed sqoop, but can't load odbc error. I already placed odbc driver in sqoop/lib location. When I run as a command it works fine.
Please help