Support Questions
Find answers, ask questions, and share your expertise

oozie sqoop action gets exit code [1] yet its corresponding mapreduce job SUCCEED

New Contributor

I am using HDP 2.3.2, with sqoop 1.4.6, oozie 4.2.0, hive 1.2.1, hadoop 2.7.1.

hive warehouse dir: /apps/hive/warehouse

I run the following sqoop commands and submit oozie job with user hdfs:hdfs

Manually run sqoop import is OK, and it did succeeded imported mysql table to hive table.

sqoop import --connect jdbc:mysql://data001.qc.com/demo27 --driver com.mysql.jdbc.Driver --username root --password root --table clb_usertoken --warehouse-dir /user/hive/warehouse --hive-import --hive-overwrite

Use oozie workflow to do the sqoop import failed.

One of the action always get this error "Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]"

And its corresponding mapreduce job is actually SUCCEEDED.

1311-job.png

I can also find the successfully processed files under /user/hive/warehouse

[hdfs@data006 import_from_mysql]$ hdfs dfs -ls -R /user/hive/warehouse
drwxr-xr-x - hdfs hdfs 0 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken
-rw-r--r-- 3 hdfs hdfs 0 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/_SUCCESS
-rw-r--r-- 3 hdfs hdfs 375228 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00000
-rw-r--r-- 3 hdfs hdfs 372955 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00001
-rw-r--r-- 3 hdfs hdfs 380625 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00002
-rw-r--r-- 3 hdfs hdfs 387143 2016-01-13 14:05 /user/hive/warehouse/clb_usertoken/part-m-00003

So it seems that the final step to import the processed result directory to hive warehouse failed.

But I can't find any useful log from oozie.log.

Please, can anyone point out what's wrong here?

------------

Attached workflow.xml (2nd action, I've deleted the unnecessary lines after.) & job.properties

security_enabled=False

nameNode=hdfs://data002.qc.com:8020

jobTracker=data002.qc.com:8050 hive2JdbcUrl=jdbc:hive2://data005.qc.com:10000/default

appRootDir=/qc/app/user_behavior_analysis

oozie.wf.application.path=${nameNode}/qc/app/user_behavior_analysis/oozie/import_from_mysql/workflow.xml

END.

13 REPLIES 13

This link only tells how to look into error . Nothing more than that unfortunately.

Mentor

yes, the error you're posting is generic, you need to use the instructions to diagnose and if still unclear, paste error from the failed task attempt, not the error you see in Oozie.

Hello @Artem Ervits

I am able to successfully run shell action and below sqoop command from my edge node -

sqoop import --connect jdbc:oracle:thin:@53.34.3.172:1542:ELSIB --username HADOOP_I --password XXXXXX --as-textfile --target-dir /org/itpgi/abagels/sqoop/BPRF_2000 --fields-terminated-by ',' --query "select * from BPRF where \$CONDITIONS" -m 1

However when its in workflow.xml as below -

<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
  <start to="source2landingZone"/>
  <action name="source2landingZone">
  <sqoop xmlns="uri:oozie:sqoop-action:0.2">
  <job-tracker>${jobTracker}</job-tracker>
  <name-node>${nameNode}</name-node>
  <command>import --connect jdbc:oracle:thin:@53.34.3.172:1542:ELSIB --username HADOOP_I --password XXXXX --as-textfile --target-dir /org/itpgi/abagels/sqoop/BPRF_2000 --fields-terminated-by ',' --query "select * from BPRF where \$CONDITIONS" -m 1</command>
  </sqoop>
  <ok to="end"/>
  <error to="kill"/>
  </action>
  <kill name="kill">
  <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>
  <end name="end"/>
</workflow-app>

I end up having below error in map job -

Log Type: stdout

Log Upload Time: Thu Apr 07 11:18:37 +0530 2016

Log Length: 198621

Showing 4096 bytes of 198621 total. Clickherefor the full log.

pTool (BaseSqoopTool.java:applyCredentialsOptions(1026)) - Setting your password on the command-line is insecure. Consider using -P instead.
3030 [main] WARN  org.apache.sqoop.SqoopOptions  - Character argument ',' has multiple characters; only the first will be used.
2016-04-07 11:18:30,296 WARN  [main] sqoop.SqoopOptions (SqoopOptions.java:toChar(1080)) - Character argument ',' has multiple characters; only the first will be used.
3030 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Error parsing arguments for import:
2016-04-07 11:18:30,296 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(304)) - Error parsing arguments for import:
3030 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: *
2016-04-07 11:18:30,296 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: *
3030 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: from
2016-04-07 11:18:30,296 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: from
3031 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: BPRF
2016-04-07 11:18:30,297 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: BPRF
3031 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: where
2016-04-07 11:18:30,297 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: where
3031 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: \$CONDITIONS"
2016-04-07 11:18:30,297 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: \$CONDITIONS"
3031 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: -m
2016-04-07 11:18:30,297 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: -m
3031 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: 1
2016-04-07 11:18:30,297 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(307)) - Unrecognized argument: 1
Intercepting System.exit(1)

Does Oozie work with different set of syntax and why its not able to recognize the import and its arguments?

Thanks,

Rohit Sureka

Based on your log, something seems to be wrong with Sqoop interpretation of the "--fields-terminated-by" option, because some characters inside XML have special meaning. Can you try to use double quotes instead of single quotes for comma, or remove that option altogether because default "--fields-terminated-by" is comma. Then, there might be a problem with "\$CONDITIONS". If that appears in the log try to remove the backslash in front of $.