Reply
Contributor
Posts: 37
Registered: ‎02-12-2016

Oozie cannot restore job

[ Edited ]

Hi all,

 

I come back with a problem with Oozie and Sqoop.

We are on Cloudera 5.7.1.

 

For information, the sqoop-metastore run on a psecific machine, and Oozie server on another one, without firewall between us.

node1: sqoop-metastore server

node2: oozie server

 

If we start a sqoop job from the cluster in Hue or our client machine, everything fine.

Sqoop start, and import from Mysql into Hive is done successfully.

 

Now we use Oozie to create a workflow with a step to do the same import.

 

So what I do:

- Run the sqoop-metastore as service on node1 on 16000 port

- Copy the sqoop-site.xml with these value from /etc/sqoop/conf/sqoop-site.xml to /etc/oozie/conf/sqoop-site.xml on node2.

 

To check if the connection to the metastore works, I launch this command from the node2

sudo -u oozie sqoop job --list

It works fine. All sqoop job configuration are displays.

 

But when we try to start a simple Oozie worflow with just this import, we have an error:

>>> Invoking Sqoop command line now >>>

3297 [uber-SubtaskRunner] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
3311 [uber-SubtaskRunner] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.6-cdh5.7.1
3549 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage  - Cannot restore job: sqoop_job_name
3549 [uber-SubtaskRunner] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage  - (No such job)
3550 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.JobTool  - I/O error performing job operation: java.io.IOException: Cannot restore missing job sqoop_job_name
	at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:256)
	at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
	at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
	at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
	at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
	at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:388)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:302)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

I don't understand why it can find the job while the ssh command from the oozie server node works...

 

In the sqoop-site.xml, we set these parameters:

<property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:hsql://remotehost:16000/sqoop</value>;
    <description>Port that this metastore should listen on.</description>
 </property>
 <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>SA</value>
    <description>The username to bind to the metastore.
    </description>
 </property>
 <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
    <description>The password to bind to the metastore.
    </description>
 </property>
 <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>true</value>
    <description>If true, allow saved passwords in the metastore.
    </description>
 </property>

The workflow configuration:

<action name="extract">
        <sqoop xmlns="uri:oozie:sqoop-action:0.4">
	  <command>
		job --exec sqoop_job_name -- --target-dir /path/to/dir --username username --password password --hive-import --hive-table database.table
	  </command>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>

We try many settings, but nothing works....

And we can't continue to works without create our workflow...

 

Somebody already encount this problem ? Or have an idea from what or where the problem come from ?

 

Thanks a lot for your help.

Contributor
Posts: 37
Registered: ‎02-12-2016

Re: Oozie cannot restore job

Update:

 

We use HA. And in the xml configuration file, we give these informations

 

<property><name>job_tracker</name><value>activenode.domain.com:8032</value></property>
        <property><name>job_xml</name><value>/cache/hive-site.xml</value></property>
        <property><name>name_node</name><value>hdfs://namenodeha</value></property>
        <property><name>oozie.use.system.libpath</name><value>true</value></property>
        <property><name>oozie.libpath</name><value>/user/oozie/share/lib/lib_20160808152436</value></property>

        <property><name>wf_path</name><value>/user/user1/test/</value></property>
        <property><name>oozie.wf.application.path</name><value>${name_node}${wf_path}/workflow-extract.xml</value></property>

        <property><name>db_user</name><value>db_login</value></property>
        <property><name>db_pass</name><value>db_password</value></property>

        <property><name>target_dir</name><value>${wf_path}/test_sqoop</value></property>

        <property><name>hive_db_schema</name><value>test</value></property>

But the "job_tracker" was set to the "backup" node int the HA configuration.

 

I try to set it with the "active" node, the job start but, go to "running" status, but the task stay in "PREP" status...

 

Our oozie workflow:

<workflow-app xmlns="uri:oozie:workflow:0.5" name="cba_1333-extract">
    <global>
	<job-tracker>${job_tracker}</job-tracker>
	<name-node>${name_node}</name-node>
	<job-xml>${job_xml}</job-xml>
    </global>


    <start to="extract"/>

    <action name="extract">
        <sqoop xmlns="uri:oozie:sqoop-action:0.4">
	  <command>
		job --exec sqoop_job_name -- --target-dir /user/user1/test/db_import --username db_user --password db_password --hive-import --hive-table db.table
	  </command>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>


	<kill name="fail">
        <message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    	</kill>
   	<end name="end"/>
</workflow-app>

 

Any idea for what the the workflow dosen't work ?

 

If we start a sqoop alone, job, it works,

If we start a an oozie, with hive job, it works...

 

Any idea ?

Contributor
Posts: 37
Registered: ‎02-12-2016

Re: Oozie cannot restore job

Forget my last try,, the yarn manager is not the this machine, so the job can't start.

 

I keep my config where the Yarn Manage is running, but kee the same problem...

Contributor
Posts: 37
Registered: ‎02-12-2016

Re: Oozie cannot restore job

I try to start our sqoop command directlry from the oozie server.

 

It doesn't work, and I have this error:

 

Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/22 20:06:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.1
16/09/22 20:06:48 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: Passed Null for map null
java.lang.IllegalArgumentException: Passed Null for map null
        at org.apache.sqoop.util.SqoopJsonUtil.getMapforJsonString(SqoopJsonUtil.java:48)
        at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:621)
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
        at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

My command:

sqoop job --exec sqoo_job_name -- --target-dir /user/user1/test/dstdir --username db_user --password db_password

I remove the part "hive import" to simply the command and it can have a result.

 

Problem still here, but with another error when I try to launch the sqoop comand only fron oozie server.

Contributor
Posts: 37
Registered: ‎02-12-2016

Re: Oozie cannot restore job

Prroblem still present...

 

Any ideas why and how to solve this problem ?

Contributor
Posts: 37
Registered: ‎02-12-2016

Re: Oozie cannot restore job

I give you more informations with value of our workflow.

 

Name	Value
db_pass	password
db_user	username
hive_db_schema	test
job_tracker	host:8032
job_xml	/cache/sqoop-site.xml
mapreduce.job.user.name	user1
name_node	hdfs://namenodeha
oozie.libpath	/user/oozie/share/lib/lib_20160808152436/
oozie.use.system.libpath	true
oozie.wf.application.path	hdfs://namenodeha/user/user1/test//workflow-extract.xml
target_dir	/user/user1/test//test_sqoop
user.name	user1
wf_path	/user/user1/test/

Maybe something wrong inside, but I don't see it...

 

Or problem still present and don't find any solution :(

 

I thinks the problem is very easy, but I don't find it.

 

On our labs cluster with 5.5.1 version, nothing specific to do to allow oozie exec sqoop actions.

All configurations are identical between us. So weird...

 

Configuation from oozie in CM ? Variable need to be declare ?

Announcements