Sqoop extra arguments supported by Netezza Connector not working

[#1]

I'm trying to import the data from Netezza table to HDFS using sqoop. It works fine without the Netezza connector specific arguments, but when I try to use --log-dir, --max-errors or any of Netezza connector specific arguments it fails.

When I run the sqoop command to import the data from Netezza table into HDFS, I get the following error:

2016-05-24 10:27:40,991 ERROR - [main:] ~ Error parsing arguments for import: (BaseSqoopTool:303)
2016-05-24 10:27:40,991 ERROR - [main:] ~ Unrecognized argument: --log-dir (BaseSqoopTool:306)
2016-05-24 10:27:40,991 ERROR - [main:] ~ Unrecognized argument: /root (BaseSqoopTool:306)

Sample command:

sqoop import \
--connect jdbc:netezza://nzserverip:5480/db1  \
--username user1 \
--password pass1 \
--direct \
--table tab1 \
--target-dir /test/tab1 \
--delete-target-dir \
--m 8  \
--log-dir "/logdir"

I'm running this on HDP 2.3 sandbox and the sqoop and hadoop versions are:

Sqoop Version:

Running Sqoop version: 1.4.6.2.3.0.0-2557 (Sqoop:92) Sqoop 1.4.6.2.3.0.0-2557 git commit id f691d38f97daa4700a1c7a7a113795de58ba77f6 Compiled by jenkins on Tue Jul 14 10:56:44 EDT 2015

Hadoop version:

Hadoop 2.7.1.2.3.0.0-2557 Subversion git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1 Compiled by jenkins on 2015-07-14T13:08Z Compiled with protoc 2.5.0

any help or guidance would be appreciate 🙂

[#2 · Accepted solution]

@Farrukh Mahmoo

Can you please try with below syntax?

sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table tab1 --target-dir /test/tab1 --delete-target-dir --m 8 --log-dir "/logdir"

[#3]

@Jitendra Yadav tried the one you mentioned plus with all the other minor variations, still the same error.

[#4]

@Farrukh Mahmood

Ok, then see if you have completed the prerequisites as mentioned by @Dave Russell

[#5]

@Farrukh Mahmood

Still thinking there might be a syntax issue with the Sqoop command, wondering if you can share the latest command you ran along with error msg?

[#6]

@Jitendra Yadav

The first command throws the error:

sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose --log-dir "/logdir"

Please set $ACCUMULO_HOME to the root of your Accumulo installation.
2016-05-24 13:30:38,094 INFO  - [main:] ~ Running Sqoop version: 1.4.6.2.3.0.0-2557 (Sqoop:92)
2016-05-24 13:30:38,117 DEBUG - [main:] ~ Enabled debug logging. (BaseSqoopTool:922)
2016-05-24 13:30:38,118 WARN  - [main:] ~ Setting your password on the command-line is insecure. Consider using -P instead. (BaseSqoopTool:1021)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Error parsing arguments for import: (BaseSqoopTool:303)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Unrecognized argument: --log-dir (BaseSqoopTool:306)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Unrecognized argument: /logdir (BaseSqoopTool:306)
2016-05-24 13:30:38,120 DEBUG - [main:] ~
Try --help for usage instructions. (Sqoop:137)

Try --help for usage instructions.
  at org.apache.sqoop.tool.ImportTool.validateOptions(ImportTool.java:1121)
  at org.apache.sqoop.Sqoop.run(Sqoop.java:133)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
  at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
  at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
  at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Try --help for usage instructions.

and once I get rid of --log-dir parameter same command works perfectly:

sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose

2016-05-24 13:32:10,432 INFO  - [main:] ~ Transferred 75.3252 KB in 28.7848 seconds (2.6168 KB/sec) (ImportJobBase:184)
2016-05-24 13:32:10,436 INFO  - [main:] ~ Retrieved 965 records. (ImportJobBase:186)
2016-05-24 13:32:10,436 DEBUG - [main:] ~ Restoring classloader: sun.misc.Launcher$AppClassLoader@4d88e490 (ClassLoaderStack:45)

(skipped the intermediate verbose output and copied the last few lines).

[#7]

@Farrukh Mahmood

This came up as an issue within HDP version, as a workaround please put extra "--" with a space as prefix along with "--log-dir" or "--max-errors". Example:

sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table tab1 --target-dir /test/tab1 --delete-target-dir --m 1 -- --log-dir /logdir/

I will ask Doc team to update the HDP netezza document.

[#8]

Hi @Farrukh Mahmood.

When you get these errors, are you using the standard version of Sqoop, or the IBM Netezza version?

As mentioned in the documentation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_dataintegration/content/netezza-connecto... you will need to put the IBM jar (named nzjdbc.jar) in the right location for it to be picked up.

The standard version of Sqoop shipped with HDP does not support those additional arguments.

Hope that helps.

[#9]

Hi Dave,

Thanks for the reply.

I think the Sqoop version doesn't need to change to connect with Netezza but the only additional thing you need is the Sqoop Connection Manager for Netezza. As per the link you sent, HDP2 already comes with the Sqoop Connection Manager for Netezza. I have verified that the Sqoop Import is using the correct Netezza connector using the verbose option.

The file nzjdbc.jar is the Netezza driver which you need in addition to Netezza connector and I have already copied it in the Sqoop lib folder. Import is working fine by using the correct Netezza Connector and Driver without these arguments but as soon as I add these arguments, the import fails with error

[#10]

Thanks @Jitendra Yadav

It's working based on your suggestion i.e by adding "-- " before the Netezza connector specific parameters.

sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose -- --max-errors 5 

and the backend query run on Netezza (one of 8 as I used --m 😎 is as follows:

CREATE EXTERNAL TABLE '/hadoop/yarn/local/usercache/root/appcache/application_1463651102463_0033/work/task_1463651102463_0033_m_000001/nzexttable-1.txt' USING (REMOTESOURCE 'JDBC' BOOLSTYLE 'T_F' CRINSTRING FALSE DELIMITER 44 ENCODING 'internal' FORMAT 'Text' INCLUDEZEROSECONDS TRUE NULLVALUE 'null' MAXERRORS 5) AS SELECT * FROM TAB1 WHERE (DATASLICEID % 😎 = 1 

Looking at the query, its taking into account the --max-error 5 argument (by adding MAXERRORS 5 in the query) .