Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop hive overwrite fails in oozie, works command line

Highlighted

Sqoop hive overwrite fails in oozie, works command line

New Contributor

So I am trying to import+overwrite a hive table with an incremental sqoop job. If I run it command line, as a regular user, everything works fine - command looks like this:

 

sqoop import --connect jdbc:oracle:thin:@sanitized.hostname.com:1521:servicesb1 --username user --password-file hdfs://nameservice1/user/user/.password_bps --table SERVICESO.ACCT --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --incremental lastmodified --check-column UPDATE_DATE --last-value "2016-02-10-22:08:18" --hive-import --hive-overwrite --hive-database service_inc --hive-table acct

boom! overwrites the existing, drops the file. However, when I call it in oozie, it looks like it kinda sees hive, but not hive overwrite?

 

Sqoop command arguments :
             import
             --connect
             jdbc:oracle:thin:@sanitized.hostname.com:1521:servicesb1
             --username
             user
             --password-file
             hdfs://nameservice1/user/user/.password_bps
             --table
             SERVICESO.ACCT
             --compress
             --compression-codec
             org.apache.hadoop.io.compress.SnappyCodec
             --incremental
             lastmodified
             --check-column
             UPDATE_DATE
             --last-value
             2016-02-10-22:08:18.0
             --hive-import
             --hive-database
             service_inc
             --hive-overwrite
             --hive-table
             acct
10975 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.ImportTool  - Error during import: --merge-key or --append is required when using --incremental lastmodified and the output directory exists.
Intercepting System.exit(1)

I have included hive-site.xml as a file in the oozie workflow, and set oozie.use.system.libpath=true

 

and it doesn't error on the hive bit -- just that it doesn't accept the hive-overwrite flag. I tried ordering the arguments differently, but got the same error. I don't know where to look next, looking for help.

 

 Running Sqoop version: 1.4.6-cdh5.5.0, on CDH5.5 (obviously), no other strange things. I do see 

 $SQOOP_CONF_DIR has not been set in the environment.

in the log as well - but when i google/read about that I find (1) everyone says that's never the problem (2) I can't see how to set it in my workflow to make it go away in any docs or anything. 

 

Let me know

1 REPLY 1

Re: Sqoop hive overwrite fails in oozie, works command line

Master Guru
Couple of questions:

1. Is your Oozie ShareLib up to date (i.e. do the Sqoop jars on it match the Sqoop version of CDH in the cluster?)

2. Have you tried using a --target to specify a dynamic output directory location, just in case that's the complaint from Sqoop?