New Contributor
Posts: 1
Registered: ‎04-03-2018

Creating / Running Sqoop Job in Shell Action Using Oozie

[ Edited ]

I'm trying to run a Sqoop job in Shell script using Oozie. Please note, that i'm using cdh5 in my local machine (vm with 12G of RAM), and HUE to build the workflow.

I created a Sqoop job that extracts data from Mysql to HDFS, in a shell script. And I run it using Oozie :

sqoop job --create testmetastore --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop -- import --connect jdbc:mysql://localhost:3306/retail_db --table EMPLOYEE --username root --password cloudera --target-dir hdfs://localhost:8020/user/cloudera/EMPLOYEES -m 1

The Job has been created (I can find it when I do a list cmd, see bellow):

sqoop job --list --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop



I even executed the job from my terminal and it works (see cmd below):

sqoop job --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop --exec testmetastore

I deleted the last imported folder, and tried to re-execute the job again but this time in Oozie (in shell action). It gives an error (see below):

18/12/23 10:47:06 INFO mapreduce.ImportJobBase: counters are unavailable. To get this information, 
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: you will need to enable the completed job store on 
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: the jobtracker with:
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: = true
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: mapreduce.jobtracker.persist.jobstatus.hours = 1
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: A jobtracker restart is required for these settings
18/12/23 10:47:06 INFO mapreduce.ImportJobBase: to take effect.
18/12/23 10:47:06 ERROR tool.ImportTool: Import failed: Import job failed!
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Here is the sqoop-site.xml:

    <description>If true, Sqoop will connect to a local metastore
      for job management when no other metastore arguments are

    <description>The connect string to use when connecting to a
      job-management metastore. If unspecified, uses ~/.sqoop/.
      You can specify a different path here.
    <description>The username to bind to the metastore.
    <description>The password to bind to the metastore.

    <description>If true, allow saved passwords in the metastore.

    <description>Path to the shared metastore database files.
    If this is not set, it will be placed in ~/.sqoop/.

    <description>Port that this metastore should listen on.

Here is the workflow.xml:

<workflow-app name="MyWorkflow" xmlns="uri:oozie:workflow:0.5">
<start to="shell-7268"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
<action name="shell-7268">
    <shell xmlns="uri:oozie:shell-action:0.1">
    <ok to="End"/>
    <error to="Kill"/>
<end name="End"/>


Here is a view from HUE: 



The reason why i'm trying to run a sqoop job in shell script from Oozie, is that I want to create a sqoop job that remenbers the last value imported so that it can increment the table, and then schedule it using Oozie. So this is just a first step test!


Can you please help ? Thanks,

Cloudera Employee
Posts: 59
Registered: ‎04-03-2017

Re: Creating / Running Sqoop Job in Shell Action Using Oozie



Can you please share the job logs from YARN page? Or you can run the below command.


--> yarn logs -applicationId <application ID> -appOwner <AppOwner>


This will tell us more info that why the job is failing.




New solutions