Member since
06-16-2017
6
Posts
0
Kudos Received
0
Solutions
05-25-2016
01:21 PM
Thanks @Jitendra Yadav It's working based on your suggestion i.e by adding "-- " before the Netezza connector specific parameters. sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose -- --max-errors 5 and the backend query run on Netezza (one of 8 as I used --m 😎 is as follows: CREATE EXTERNAL TABLE '/hadoop/yarn/local/usercache/root/appcache/application_1463651102463_0033/work/task_1463651102463_0033_m_000001/nzexttable-1.txt' USING (REMOTESOURCE 'JDBC' BOOLSTYLE 'T_F' CRINSTRING FALSE DELIMITER 44 ENCODING 'internal' FORMAT 'Text' INCLUDEZEROSECONDS TRUE NULLVALUE 'null' MAXERRORS 5) AS SELECT * FROM TAB1 WHERE (DATASLICEID % 8) = 1 Looking at the query, its taking into account the --max-error 5 argument (by adding MAXERRORS 5 in the query) .
... View more
05-24-2016
03:56 PM
@Jitendra Yadav The first command throws the error: sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose --log-dir "/logdir"
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
2016-05-24 13:30:38,094 INFO - [main:] ~ Running Sqoop version: 1.4.6.2.3.0.0-2557 (Sqoop:92)
2016-05-24 13:30:38,117 DEBUG - [main:] ~ Enabled debug logging. (BaseSqoopTool:922)
2016-05-24 13:30:38,118 WARN - [main:] ~ Setting your password on the command-line is insecure. Consider using -P instead. (BaseSqoopTool:1021)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Error parsing arguments for import: (BaseSqoopTool:303)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Unrecognized argument: --log-dir (BaseSqoopTool:306)
2016-05-24 13:30:38,119 ERROR - [main:] ~ Unrecognized argument: /logdir (BaseSqoopTool:306)
2016-05-24 13:30:38,120 DEBUG - [main:] ~
Try --help for usage instructions. (Sqoop:137)
Try --help for usage instructions.
at org.apache.sqoop.tool.ImportTool.validateOptions(ImportTool.java:1121)
at org.apache.sqoop.Sqoop.run(Sqoop.java:133)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Try --help for usage instructions. and once I get rid of --log-dir parameter same command works perfectly: sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table TAB1 --target-dir /test/tab1 --delete-target-dir --m 8 --verbose 2016-05-24 13:32:10,432 INFO - [main:] ~ Transferred 75.3252 KB in 28.7848 seconds (2.6168 KB/sec) (ImportJobBase:184)
2016-05-24 13:32:10,436 INFO - [main:] ~ Retrieved 965 records. (ImportJobBase:186)
2016-05-24 13:32:10,436 DEBUG - [main:] ~ Restoring classloader: sun.misc.Launcher$AppClassLoader@4d88e490 (ClassLoaderStack:45) (skipped the intermediate verbose output and copied the last few lines).
... View more
05-24-2016
11:17 AM
Hi Dave, Thanks for the reply. I think the Sqoop version doesn't need to change to connect with Netezza but the only additional thing you need is the Sqoop Connection Manager for Netezza. As per the link you sent, HDP2 already comes with the Sqoop Connection Manager for Netezza. I have verified that the Sqoop Import is using the correct Netezza connector using the verbose option. The file nzjdbc.jar is the Netezza driver which you need in addition to Netezza connector and I have already copied it in the Sqoop lib folder. Import is working fine by using the correct Netezza Connector and Driver without these arguments but as soon as I add these arguments, the import fails with error
... View more
05-24-2016
11:06 AM
@Jitendra Yadav tried the one you mentioned plus with all the other minor variations, still the same error.
... View more
05-24-2016
10:35 AM
I'm trying to import the data from Netezza table to HDFS using sqoop. It works fine without the Netezza connector specific arguments, but when I try to use --log-dir, --max-errors or any of Netezza connector specific arguments it fails. When I run the sqoop command to import the data from Netezza table into HDFS, I get the following error: 2016-05-24 10:27:40,991 ERROR - [main:] ~ Error parsing arguments for import: (BaseSqoopTool:303)
2016-05-24 10:27:40,991 ERROR - [main:] ~ Unrecognized argument: --log-dir (BaseSqoopTool:306)
2016-05-24 10:27:40,991 ERROR - [main:] ~ Unrecognized argument: /root (BaseSqoopTool:306) Sample command: sqoop import \
--connect jdbc:netezza://nzserverip:5480/db1 \
--username user1 \
--password pass1 \
--direct \
--table tab1 \
--target-dir /test/tab1 \
--delete-target-dir \
--m 8 \
--log-dir "/logdir" I'm running this on HDP 2.3 sandbox and the sqoop and hadoop versions are: Sqoop Version: Running Sqoop version: 1.4.6.2.3.0.0-2557 (Sqoop:92)
Sqoop 1.4.6.2.3.0.0-2557
git commit id f691d38f97daa4700a1c7a7a113795de58ba77f6
Compiled by jenkins on Tue Jul 14 10:56:44 EDT 2015 Hadoop version: Hadoop 2.7.1.2.3.0.0-2557
Subversion git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1
Compiled by jenkins on 2015-07-14T13:08Z
Compiled with protoc 2.5.0 any help or guidance would be appreciate 🙂
... View more
Labels:
- Labels:
-
Apache Sqoop