Sqoop to import data to Hive through oozie shell action fails

[#1]

Hi,

I am running a oozie shell action to run a sqoop command to import data to Hive. When I run the sqoop command directly, it works fine, but when I run it through oozie shell action, it aborts with

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Based on this link,

https://community.hortonworks.com/questions/25121/oozie-execute-sqoop-falls.html#answer-25290

I have added hive-site.xml also using <file> tag in oozie shell action and also based on other link I have added

export HIVE_CONF_DIR=`pwd` 

before running the sqoop command. But nothing worked. When I add full hive-site.xml it resulted in the same error above, when I added just the important properties mentioned in this link http://ingest.tips/2014/11/27/how-to-oozie-sqoop-hive/, I get this error

FAILED: IllegalStateException Unxpected Exception thrown: Unable to fetch table XYZ. java.net.SocketException: Connection resetFailing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Both the times, the sqoop command successfully creates the file in target-directory but fails while loading this data to hive.

Hadoop cluster is kerberos enabled. I have a kinit done before submitting the workflow and also kinit is done again inside the oozie shell action.

Can someone please throw some light on how to fix this one? below is the sqoop command used.

Sqoop command:sqoop import \
--connect "jdbc:teradata://${server}/database=${db},logmech=ldap" \
--driver "com.teradata.jdbc.TeraDriver" \
--table "XYZ" \
--split-by "col1" \
--hive-import \
--delete-target-dir \
--target-dir "/user/test/" \
--hive-table "default.XYZ" \
--username "terauser" \
--password tdpwd \
--where "${CONDITION}" \
--m 2 \
--fetch-size 1000 \
--hive-drop-import-delims \
--fields-terminated-by '\001' \
--lines-terminated-by '\n' \
--null-string '\\N' \
--null-non-string '\\N'

[#2]

Hello,

Since it's a exit code [1] error, you'll need to get more information about what is causing the exit code.

Find out the application ID of the job launched:

yarn application -list -appTypes ALL

su - as the user used to launch the job and run:

yarn logs -applicationId <application ID>

[#3]

Hi, I checked the logs. No information found as to why the script aborted. This is all is shown in the log.

INFO hive.HiveImport: Loading uploaded data into Hive
WARN conf.HiveConf: HiveConf of name hive.metastore.pre-event.listeners does not exist
WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist
Logging initialized using configuration in jar:file:/grid/8/hadoop/yarn/local/filecache/5470/hive-common-1.2.1.2.3.4.0-3485.jar!/hive-log4j.properties
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

[#4 · Accepted solution]

When I added the hive-site.xml first, I missed a few properties, now I added all properties mentioned by @allen huang

in this link

https://community.hortonworks.com/questions/25121/oozie-execute-sqoop-falls.html#answer-25291

So even if the sqoop is called using ooziee shell action, I had to add hive-site.xml with properties mentioned by Allen.

Thank you Allen :). My script is working fine now.