Reply
New Contributor
Posts: 3
Registered: ‎05-04-2015

Sqoop hive overwrite fails in oozie, works command line

So I am trying to import+overwrite a hive table with an incremental sqoop job. If I run it command line, as a regular user, everything works fine - command looks like this:

 

sqoop import --connect jdbc:oracle:thin:@sanitized.hostname.com:1521:servicesb1 --username user --password-file hdfs://nameservice1/user/user/.password_bps --table SERVICESO.ACCT --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --incremental lastmodified --check-column UPDATE_DATE --last-value "2016-02-10-22:08:18" --hive-import --hive-overwrite --hive-database service_inc --hive-table acct

boom! overwrites the existing, drops the file. However, when I call it in oozie, it looks like it kinda sees hive, but not hive overwrite?

 

Sqoop command arguments :
             import
             --connect
             jdbc:oracle:thin:@sanitized.hostname.com:1521:servicesb1
             --username
             user
             --password-file
             hdfs://nameservice1/user/user/.password_bps
             --table
             SERVICESO.ACCT
             --compress
             --compression-codec
             org.apache.hadoop.io.compress.SnappyCodec
             --incremental
             lastmodified
             --check-column
             UPDATE_DATE
             --last-value
             2016-02-10-22:08:18.0
             --hive-import
             --hive-database
             service_inc
             --hive-overwrite
             --hive-table
             acct
10975 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.ImportTool  - Error during import: --merge-key or --append is required when using --incremental lastmodified and the output directory exists.
Intercepting System.exit(1)

I have included hive-site.xml as a file in the oozie workflow, and set oozie.use.system.libpath=true

 

and it doesn't error on the hive bit -- just that it doesn't accept the hive-overwrite flag. I tried ordering the arguments differently, but got the same error. I don't know where to look next, looking for help.

 

 Running Sqoop version: 1.4.6-cdh5.5.0, on CDH5.5 (obviously), no other strange things. I do see 

 $SQOOP_CONF_DIR has not been set in the environment.

in the log as well - but when i google/read about that I find (1) everyone says that's never the problem (2) I can't see how to set it in my workflow to make it go away in any docs or anything. 

 

Let me know

Posts: 1,836
Kudos: 415
Solutions: 295
Registered: ‎07-31-2013

Re: Sqoop hive overwrite fails in oozie, works command line

Couple of questions:

1. Is your Oozie ShareLib up to date (i.e. do the Sqoop jars on it match the Sqoop version of CDH in the cluster?)

2. Have you tried using a --target to specify a dynamic output directory location, just in case that's the complaint from Sqoop?
Announcements