oozie sqoop action gets exit code [1] yet its corresponding mapreduce job SUCCEED

[#1]

I am using HDP 2.3.2, with sqoop 1.4.6, oozie 4.2.0, hive 1.2.1, hadoop 2.7.1.

hive warehouse dir: /apps/hive/warehouse

I run the following sqoop commands and submit oozie job with user hdfs:hdfs

Manually run sqoop import is OK, and it did succeeded imported mysql table to hive table.

sqoop import --connect jdbc:mysql://data001.qc.com/demo27 --driver com.mysql.jdbc.Driver --username root --password root --table clb_usertoken --warehouse-dir /user/hive/warehouse --hive-import --hive-overwrite

Use oozie workflow to do the sqoop import failed.

One of the action always get this error "Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]"

And its corresponding mapreduce job is actually SUCCEEDED.

I can also find the successfully processed files under /user/hive/warehouse

[hdfs@data006 import_from_mysql]$ hdfs dfs -ls -R /user/hive/warehouse

drwxr-xr-x - hdfs hdfs 0 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken

-rw-r--r-- 3 hdfs hdfs 0 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/_SUCCESS

-rw-r--r-- 3 hdfs hdfs 375228 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00000

-rw-r--r-- 3 hdfs hdfs 372955 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00001

-rw-r--r-- 3 hdfs hdfs 380625 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00002

-rw-r--r-- 3 hdfs hdfs 387143 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00003

So it seems that the final step to import the processed result directory to hive warehouse failed.

But I can't find any useful log from oozie.log.

Please, can anyone point out what's wrong here?

------------

Attached workflow.xml (2nd action, I've deleted the unnecessary lines after.) & job.properties

security_enabled=False

nameNode=hdfs://data002.qc.com:8020

jobTracker=data002.qc.com:8050 hive2JdbcUrl=jdbc:hive2://data005.qc.com:10000/default

appRootDir=/qc/app/user_behavior_analysis

oozie.wf.application.path=${nameNode}/qc/app/user_behavior_analysis/oozie/import_from_mysql/workflow.xml

END.

[#2]

I remember similar problems, the log error was hidden somewhere either in the launcher log or in the oozie job log. ( if you have hue it is a bit clicking it could either be in the action log or the main job log ). My problem was that the hive import of sqoop requires the hive client to be installed on every datanode since sqoop simply uses the hive command under the cover. I ended up splitting up the job into two actions one for sqoop to load and a second hive2 action for the import.

[#3]

thanks this can be another working workaround solution.

[#4]

@Hao Ji it means the user has permission problems moving the final results to hive. Here's a working example I built, https://github.com/dbist/oozie/tree/master/apps/sqoopshell replace my sqoop command in script.sh with yours.

[#5]

Thanks.

Currently I am using ssh action to call sqoop import as workaround.

[#6]

I'm using HDP 2.3.2 sandbox. To make sqoop from oozie run, I needed:

- set port forwarding for 10020 in sandbox to avoid connection refuse issue

- copy oracle jdbc drive to HDFS /user/oozie/share/lib/lib_20151027124452/sqoop directory

Now, I can run sqoop list-tables or list-databases command; however, sqoop import fails with both HDFS and Hive. Since it just return error code like in this article, I don't know what's wrong exactly. Would you have any suggestion kindly?

[#7]

Hello All. I am facing similar problem .

A little background , Cluster is running with HDP 2.3.2 kerberos enabled . Now when running sqoop import command alone it works fine . Also Oozie is working well with sqoop list table, However when Oozie has below workflow -

<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
  <start to="source2landingZone"/>
  <action name="source2landingZone">
  <sqoop xmlns="uri:oozie:sqoop-action:0.2">
  <job-tracker>${jobTracker}</job-tracker>
  <name-node>${nameNode}</name-node>
  <command>sqoop import --connect jdbc:oracle:thin:@53.34.3.172:1542:ELSIB --username HADOOP_I --password xxxxx --as-textfile --target-dir /org/itpgi/abagels/sqoop/BPRF_2 --fields-terminated-by ',' --query "select * from BPRF where \$CONDITIONS" -m 1</command>
  </sqoop>
  <ok to="end"/>
  <error to="kill"/>
  </action>
  <kill name="kill">
  <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>
  <end name="end"/>
</workflow-app>

it fails -

with error -

Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Job is successful though. with succeeded status and no bad logs in RM.

Any help is highly appreciated.

Thanks,

Rohit Sureka

[#8]

@Artem Ervits @Adam Whitter

I tried also with Shell Action however it fails with same error code -

Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Could you please help me ?

Thank you,

Rohit Sureka

[#9]

Please use this to diagnose your problem https://community.hortonworks.com/articles/9148/troubleshooting-an-oozie-flow.html

[#10]

Thank you very much.