Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to use sqoop on cloudera CDH5

Solved Go to solution

Unable to use sqoop on cloudera CDH5

Contributor

Hi team,

 

I am getting below error while trying to use sqoop syntax. Unable to use sqoop on cloudera CDH5

 

root@blrrndtipamo02:/opt/cloudera/parcels/CDH/lib/sqoop/bin# sqoop version
Error: /usr/lib/sqoop/../hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.

 

 

root@blrrndtipamo02:/opt/cloudera/parcels/CDH/lib/sqoop/bin# sqoop help
Error: /usr/lib/sqoop/../hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.

 

Please suggest

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Unable to use sqoop on cloudera CDH5

Contributor

Issue fixed with below steps I performed.

 

  1. Download derby derby client from db-derby-10.11.1.1-bin.zip
  2. Extract and copy derby.jar and derbyclient.jar from zip file to /var/lib/sqoop2
  3. Copy derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/ as well
  4. Delete /opt/cloudera/parcels/CDH-<version>/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby​-<version>.jar soft link.
  5. Make /opt/cloudera/parcels/CDH-<version>/l​ib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar to /opt/cloudera/parcels/CDH-<version>/jars/derby.jar

Command for soft link:

ln -s /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/sqoop2/webapps/sqoop/WEB-INF/lib/derby.jar /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/derby.jar

11 REPLIES 11

Re: Unable to use sqoop on cloudera CDH5

Rising Star

What does 'hadoop version' return on that host?

Are you deploying HDFS or YARN gateway role to that host? See http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_managing_roles.ht...

Re: Unable to use sqoop on cloudera CDH5

Contributor

CDH5 is installed

 

The above issue got fixed, now I get below issue while  trying to run sqoop syntax.

 

sqoop import --verbose --connect jdbc:oracle://hostname.net:15012/mid --username srpnbhat --password Bnmae3@ --table SRP_RUN --target-dir /user/hive/warehouse/nag

 

 

<script src="res://instant-message.dll/2438" type="text/javascript"></script>

15/05/13 11:20:16 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 11:20:16 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 11:20:16 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)

 

Please suggest if we are missing anything here

Highlighted

Re: Unable to use sqoop on cloudera CDH5

Rising Star

Have you installed the Oracle JDBC driver per the following instructions?

 

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_jdbc_driver_inst...

Re: Unable to use sqoop on cloudera CDH5

Contributor

Hi

 

As per steps on above link  I did not have dir as /usr/lib/sqoop/lib. I create dir sqoop/lib and copied the ojdbc6_g.jar file

 

Still I get same error 

 

15/05/13 13:10:51 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.3
15/05/13 13:10:51 DEBUG tool.BaseSqoopTool: Enabled debug logging.
15/05/13 13:10:51 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 13:10:51 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
15/05/13 13:10:51 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 13:10:51 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:
15/05/13 13:10:51 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
15/05/13 13:10:51 INFO manager.SqlManager: Using default fetchSize of 1000
15/05/13 13:10:51 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@8297b3a
15/05/13 13:10:51 INFO tool.CodeGenTool: Beginning code generation
15/05/13 13:10:51 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 13:10:51 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 13:10:51 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Re: Unable to use sqoop on cloudera CDH5

Rising Star

Can you rename the jar file to ojdbc6.jar and give it a try?

Re: Unable to use sqoop on cloudera CDH5

Contributor

Still same error

 

 

root@blrrndtipamo01:/usr/lib/sqoop/lib# mv ojdbc6_g.jar ojdbc6.jar
root@blrrndtipamo01:/usr/lib/sqoop/lib# ls
ojdbc6.jar
root@blrrndtipamo01:/usr/lib/sqoop/lib# sudo sqoop import --verbose --connect jdbc:oracle://hostname.net:15012/mid --username srpnbhat --password xxxxxxx--table SRP_RUN --target-dir /user/hive/warehouse/nag
Warning: /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/05/13 15:33:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.3
15/05/13 15:33:56 DEBUG tool.BaseSqoopTool: Enabled debug logging.
15/05/13 15:33:56 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
15/05/13 15:33:56 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
15/05/13 15:33:56 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
15/05/13 15:33:56 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:
15/05/13 15:33:56 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
15/05/13 15:33:56 INFO manager.SqlManager: Using default fetchSize of 1000
15/05/13 15:33:56 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@8297b3a
15/05/13 15:33:56 INFO tool.CodeGenTool: Beginning code generation
15/05/13 15:33:56 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 15:33:56 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM SRP_RUN t WHERE 1=0
15/05/13 15:33:56 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:286)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Re: Unable to use sqoop on cloudera CDH5

Rising Star

Is the permission configured correctly?

 

chown sqoop:sqoop /var/lib/sqoop 
chmod 755 /var/lib/sqoop

Re: Unable to use sqoop on cloudera CDH5

Contributor

Yes, prerequities already taken care.

 

premission check

drwxr-xr-x 2 sqoop sqoop 4.0K May 13 13:42 sqoop

Re: Unable to use sqoop on cloudera CDH5

Contributor

Please let me know if anything to be taken care in configuration files...such as hdfs setup or Hive.....namenode format

 

 

CDH5 on Debain linux OS

 

Expecting the sqoop to import oracle data into HDFS. Please suggest