question Re: Unable to use sqoop on cloudera CDH5 in Archives of Support Questions (Read Only)

Unable to use sqoop on cloudera CDH5

Guru007 — Fri, 16 Sep 2022 09:28:38 GMT

Hi team,

I am getting below error while trying to use sqoop syntax. Unable to use sqoop on cloudera CDH5

root@blrrndtipamo02:/opt/cloudera/parcels/CDH/lib/sqoop/bin# sqoop version
Error: /usr/lib/sqoop/../hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.

root@blrrndtipamo02:/opt/cloudera/parcels/CDH/lib/sqoop/bin# sqoop help
Error: /usr/lib/sqoop/../hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.

Please suggest

Re: Unable to use sqoop on cloudera CDH5

dice — Wed, 13 May 2015 05:22:02 GMT

What does 'hadoop version' return on that host?

Are you deploying HDFS or YARN gateway role to that host? See http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_managing_roles.html?scroll=xd_583c10bfdbd326ba-3ca24a24-13d80143249--7f3c__section_scv_ywt_cn.

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Wed, 13 May 2015 06:17:04 GMT

CDH5 is installed

The above issue got fixed, now I get below issue while trying to run sqoop syntax.

sqoop import --verbose --connect jdbc:oracle://hostname.net:15012/mid --username srpnbhat --password Bnmae3@ --table SRP_RUN --target-dir /user/hive/warehouse/nag

15/05/13 11:20:16 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 11:20:16 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 11:20:16 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)

Please suggest if we are missing anything here

Re: Unable to use sqoop on cloudera CDH5

dice — Wed, 13 May 2015 06:52:00 GMT

Have you installed the Oracle JDBC driver per the following instructions?

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_jdbc_driver_install.html?scroll=topic_13_7

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Wed, 13 May 2015 07:48:33 GMT

As per steps on above link I did not have dir as /usr/lib/sqoop/lib. I create dir sqoop/lib and copied the ojdbc6_g.jar file

Still I get same error

15/05/13 13:10:51 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.3
15/05/13 13:10:51 DEBUG tool.BaseSqoopTool: Enabled debug logging.
15/05/13 13:10:51 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 13:10:51 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
15/05/13 13:10:51 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 13:10:51 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:
15/05/13 13:10:51 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
15/05/13 13:10:51 INFO manager.SqlManager: Using default fetchSize of 1000
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@8297b3a
15/05/13 13:10:51 INFO tool.CodeGenTool: Beginning code generation
15/05/13 13:10:51 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 13:10:51 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 13:10:51 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Re: Unable to use sqoop on cloudera CDH5

dice — Wed, 13 May 2015 08:53:34 GMT

Can you rename the jar file to ojdbc6.jar and give it a try?

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Wed, 13 May 2015 10:05:12 GMT

Still same error

root@blrrndtipamo01:/usr/lib/sqoop/lib# mv ojdbc6_g.jar ojdbc6.jar
root@blrrndtipamo01:/usr/lib/sqoop/lib# ls
ojdbc6.jar
root@blrrndtipamo01:/usr/lib/sqoop/lib# sudo sqoop import --verbose --connect jdbc:oracle://hostname.net:15012/mid --username srpnbhat --password xxxxxxx--table SRP_RUN --target-dir /user/hive/warehouse/nag
Warning: /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/05/13 15:33:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.3
15/05/13 15:33:56 DEBUG tool.BaseSqoopTool: Enabled debug logging.
15/05/13 15:33:56 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 15:33:56 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
15/05/13 15:33:56 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 15:33:56 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:
15/05/13 15:33:56 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
15/05/13 15:33:56 INFO manager.SqlManager: Using default fetchSize of 1000
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@8297b3a
15/05/13 15:33:56 INFO tool.CodeGenTool: Beginning code generation
15/05/13 15:33:56 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 15:33:56 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 15:33:56 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Re: Unable to use sqoop on cloudera CDH5

dice — Wed, 13 May 2015 10:10:48 GMT

Is the permission configured correctly?

chown sqoop:sqoop /var/lib/sqoop 
chmod 755 /var/lib/sqoop

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Wed, 13 May 2015 11:47:11 GMT

Yes, prerequities already taken care.

premission check

drwxr-xr-x 2 sqoop sqoop 4.0K May 13 13:42 sqoop

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Thu, 14 May 2015 05:32:00 GMT

Please let me know if anything to be taken care in configuration files...such as hdfs setup or Hive.....namenode format

CDH5 on Debain linux OS

Expecting the sqoop to import oracle data into HDFS. Please suggest

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Thu, 21 May 2015 04:37:32 GMT

Yes, permission all fine.

Checked jar file is placed in /var/lib/sqoop path.

I am using Debain Wheezy build with CDH5.3.3 parcels version. I do not see only sqoop as it shows sqoop 2 alone as in screen shoft below. However, it is doesn't work.

Re: Unable to use sqoop on cloudera CDH5

Guru007 — Fri, 21 Aug 2015 10:04:21 GMT

Issue fixed with below steps I performed.

Download derby derby client from db-derby-10.11.1.1-bin.zip
Extract and copy derby.jar and derbyclient.jar from zip file to /var/lib/sqoop2
Copy derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/ as well
Delete /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby-<version>.jar soft link.
Make /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/derby.jar

Command for soft link:

ln -s /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/derby.jar