Member since
05-13-2016
19
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1276 | 07-14-2016 01:53 AM | |
7851 | 06-15-2016 02:50 PM |
09-06-2016
06:36 AM
Thanks Constantin, This issue was resolved as updated. FYI, the SQOOP_HOME variable has always been empty in any user. I think somehow sqoop, as a part of a default HDP installation, is configured to locate lib files under /usr/hdp/current/sqoop-client/lib/ regardless of the variable value.
... View more
09-06-2016
06:08 AM
Thank you vranganathan. This issue is resolved. As you pointed out there was a problem with the ojdbc6.jar file itself. downloading directly via the web url of the file neither download the complete file, nor spit error. I guess this is because of the web authentication oracle requires. After manually downloading the file after authentication and transferring the file into the sqoop host did the trick. Thanks again.
... View more
09-06-2016
02:57 AM
Thanks Constantin, There is no such directory /usr/lib/sqoop/lib. I have been restarted the service as well as the entire cluster several times along the way. So it shouldn't be the problem. Postgresql driver, FYI, worked immediately after placing into the directory above without service restart. [root@ip-172-31-29-145 lib]# cd sqoop
-bash: cd: sqoop: No such file or directory
[root@ip-172-31-29-145 lib]# pwd
/usr/lib
[root@ip-172-31-29-145 lib]# cd sqoop
-bash: cd: sqoop: No such file or directory
Sure, I will certainly vote and accept, if there is any actual benefit in
the answers. In the mean time I deeply appreciate everyone involved in
the conversation.
... View more
09-06-2016
12:58 AM
Thanks vranganathan, As you can see in the below, ojdbc6.jar file has the same privilege(644 owner root) as all other preloaded drivers. I took the exact the same steps for the postgresql driver, and sqoop against postgresql works fine. only oracle doesn't work. [root@ip-172-31-29-145 lib]# ls -ltr
total 16164
-rw-r--r-- 1 root root 1245776 Apr 25 18:02 snappy-java-1.0.5.jar
-rw-r--r-- 1 root root 60530 Apr 25 18:02 commons-logging-1.1.1.jar
-rw-r--r-- 1 root root 266289 Apr 25 18:02 commons-jexl-2.1.1.jar
-rw-r--r-- 1 root root 770013 Apr 25 18:02 jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 1 root root 1741973 Apr 25 18:02 kite-data-mapreduce-1.0.0.jar
-rw-r--r-- 1 root root 195678 Apr 25 18:02 jackson-core-2.3.1.jar
-rw-r--r-- 1 root root 224135 Apr 25 18:02 ant-contrib-1.0b3.jar
-rw-r--r-- 1 root root 94608 Apr 25 18:02 xz-1.0.jar
-rw-r--r-- 1 root root 241259 Apr 25 18:02 commons-compress-1.4.1.jar
-rw-r--r-- 1 root root 41899 Apr 25 18:03 parquet-avro-1.4.1.jar
-rw-r--r-- 1 root root 902327 Apr 25 18:03 jackson-databind-2.3.1.jar
-rw-r--r-- 1 root root 1015097 Apr 25 18:03 parquet-jackson-1.4.1.jar
-rw-r--r-- 1 root root 2113186 Apr 25 18:03 kite-data-core-1.0.0.jar
-rw-r--r-- 1 root root 108990 Apr 25 18:03 commons-io-1.4.jar
-rw-r--r-- 1 root root 1774917 Apr 25 18:03 kite-data-hive-1.0.0.jar
-rw-r--r-- 1 root root 10378 Apr 25 18:03 parquet-generator-1.4.1.jar
-rw-r--r-- 1 root root 466100 Apr 25 18:03 parquet-format-2.0.0.jar
-rw-r--r-- 1 root root 19816 Apr 25 18:03 opencsv-2.3.jar
-rw-r--r-- 1 root root 706015 Apr 25 18:03 hsqldb-1.8.0.10.jar
-rw-r--r-- 1 root root 36427 Apr 25 18:03 ant-eclipse-1.0-jvm1.2.jar
-rw-r--r-- 1 root root 58131 Apr 25 18:03 commons-codec-1.4.jar
-rw-r--r-- 1 root root 713848 Apr 25 18:03 parquet-column-1.4.1.jar
-rw-r--r-- 1 root root 11359 Apr 25 18:03 parquet-common-1.4.1.jar
-rw-r--r-- 1 root root 400090 Apr 25 18:03 avro-1.7.5.jar
-rw-r--r-- 1 root root 152255 Apr 25 18:03 parquet-hadoop-1.4.1.jar
-rw-r--r-- 1 root root 272869 Apr 25 18:03 parquet-encoding-1.4.1.jar
-rw-r--r-- 1 root root 1740983 Apr 25 18:04 kite-hadoop-compatibility-1.0.0.jar
-rw-r--r-- 1 root root 25477 Apr 25 18:04 slf4j-api-1.6.1.jar
-rw-r--r-- 1 root root 34047 Apr 25 18:04 jackson-annotations-2.3.0.jar
-rw-r--r-- 1 root root 170228 Apr 25 18:04 avro-mapred-1.7.5-hadoop2.jar
-rw-r--r-- 1 root root 229445 Apr 25 18:04 jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 root root 29532 Apr 25 18:04 paranamer-2.3.jar
-rw-r--r-- 1 root root 579785 Sep 2 14:21 postgresql-9.2-1002.jdbc4.jar
-rw-r--r-- 1 root root 2123 Sep 2 14:31 ojdbc6.jar
-rw-r--r-- 1 root root 20570 Sep 2 16:24 optim_items.java
lrwxrwxrwx 1 root root 40 Sep 5 09:50 mysql-connector-java.jar -> /usr/share/java/mysql-connector-java.jar
[root@ip-172-31-29-145 lib]#
... View more
09-05-2016
12:25 AM
As of my original post, I have already uploaded those jdbc jar file in the library directory( /usr/hdp/current/sqoop-client/lib/) and sqoop against oracle still doesn't work.
... View more
09-02-2016
06:34 AM
Thanks vranganathan, I believe /usr/hdp/current/sqoop-client/lib/, specified above is the lib directory. Is it not? This is vanilla installation of HDP2.4, and I haven't done any additional sqoop configuration. Regards,
... View more
09-02-2016
05:28 AM
2 Kudos
Hi All, I am having an issue with sqoop against an Oracle database. HDP2.4. Oracle 11 XE in AWS RDS. [root@ip-172-31-29-111 lib]# sqoop list-tables --connect jdbc:oracle:thin:@//dbip:dbport/dbname --username myusername -password mypassword
Warning: /usr/hdp/2.4.2.0-258/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/09/02 15:23:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258
16/09/02 15:23:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/09/02 15:23:54 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
16/09/02 15:23:54 INFO manager.SqlManager: Using default fetchSize of 1000
16/09/02 15:23:54 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:287)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.OracleManager.listTables(OracleManager.java:745)
at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
The equivalent against a PostgreSQL works fine. The below is the list of jar files under sqoop library directory. [root@ip-172-31-29-111 lib]# ls /usr/hdp/current/sqoop-client/lib/
ant-contrib-1.0b3.jar commons-jexl-2.1.1.jar jackson-mapper-asl-1.9.13.jar opencsv-2.3.jar parquet-generator-1.4.1.jar
ant-eclipse-1.0-jvm1.2.jar commons-logging-1.1.1.jar kite-data-core-1.0.0.jar paranamer-2.3.jar parquet-hadoop-1.4.1.jar
avro-1.7.5.jar hsqldb-1.8.0.10.jar kite-data-hive-1.0.0.jar parquet-avro-1.4.1.jar parquet-jackson-1.4.1.jar
avro-mapred-1.7.5-hadoop2.jar
jackson-annotations-2.3.0.jar kite-data-mapreduce-1.0.0.jar
parquet-column-1.4.1.jar postgresql-9.2-1002.jdbc4.jar
commons-codec-1.4.jar
jackson-core-2.3.1.jar kite-hadoop-compatibility-1.0.0.jar
parquet-common-1.4.1.jar slf4j-api-1.6.1.jar
commons-compress-1.4.1.jar
jackson-core-asl-1.9.13.jar mysql-connector-java.jar
parquet-encoding-1.4.1.jar snappy-java-1.0.5.jar
commons-io-1.4.jar jackson-databind-2.3.1.jar ojdbc6.jar Earlier I have downloaded and copied the oracle jdbc connector jar file from the below Oracle page. http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html Please advise the solution to avoid the error. Thanks,
... View more
Labels:
- Labels:
-
Apache Sqoop
07-15-2016
03:02 AM
Sorry, I hate to repeat this, but this issue came back again. I am not trying to fool anyone here. Sorry for the misleading clue above. This time no change has been made but the cluster restart. Same error. Restarting doesn't solve the problem. This issue is so intermittent so very hard to troubleshoot.
... View more
07-14-2016
03:45 AM
Sorry. I hate to repeat this, but this issue is gone. Only change I made as per HBase is the below ambari metrics parameter values following Ambari's recommendation. hbase_master_xmn_size -> 384
hbase_master_heapsize -> 1115 Upon cluster restart, Hbase came up alright.
... View more
07-14-2016
01:53 AM
Resolved. I followed the below document to install python-pip manually in another way: http://www.liquidweb.com/kb/how-to-install-pip-on-centos-7/ [root@ip-172-31-25-3 ~]# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
--2016-07-14 11:49:14-- https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
Resolving dl.fedoraproject.org... 209.132.181.25, 209.132.181.26, 209.132.181.27, ...
Connecting to dl.fedoraproject.org|209.132.181.25|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14540 (14K) [application/x-rpm]
Saving to: “epel-release-latest-6.noarch.rpm”
100%[======================================================================================================================================>] 14,540 90.8K/s in 0.2s
2016-07-14 11:49:15 (90.8 KB/s) - “epel-release-latest-6.noarch.rpm” saved [14540/14540]
[root@ip-172-31-25-3 ~]# ll
total 20
-rw-r--r-- 1 root root 14540 Nov 6 2012 epel-release-latest-6.noarch.rpm
drwxr-xr-x 3 root root 4096 May 30 13:11 guardium_work
[root@ip-172-31-25-3 ~]# sudo rpm -Uvh epel-release-latest-6.noarch.rpm
warning: epel-release-latest-6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing... ########################################### [100%]
1:epel-release ########################################### [100%]
[root@ip-172-31-25-3 ~]# yum -y install python-pip
Loaded plugins: amazon-id, rhui-lb, security
epel/metalink | 2.7 kB 00:00
epel | 4.3 kB 00:00
epel/primary_db | 5.8 MB 00:00
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-pip.noarch 0:7.1.0-1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================
Installing:
python-pip noarch 7.1.0-1.el6 epel 1.5 M
Transaction Summary
================================================================================================================================================================================
Install 1 Package(s)
Total download size: 1.5 M
Installed size: 6.6 M
Downloading Packages:
python-pip-7.1.0-1.el6.noarch.rpm | 1.5 MB 00:00
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
Userid : EPEL (6) <epel@fedoraproject.org>
Package: epel-release-6-8.noarch (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Installing : python-pip-7.1.0-1.el6.noarch 1/1
Verifying : python-pip-7.1.0-1.el6.noarch 1/1
Installed:
python-pip.noarch 0:7.1.0-1.el6
Complete!
[root@ip-172-31-25-3 ~]#
After installing python-pip via yum, reinstalled zeppelin and it went fine. Hope this helps.
... View more