Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive and Sqoop Integration which are running on different machine

Highlighted

Hive and Sqoop Integration which are running on different machine

New Contributor

Hi ,

I am running Hive on hadoop-master node and Sqoop on one of the slave node. I am trying to create a table inside hive using sqoop by executing below command.

Questions:

1.How to connect remote hive server using sqoop ?

2. What is the best practice when we have both hive and sqoop in the architecture ?

/usr/lib/sqoop/bin/sqoop import \
--connect jdbc:mysql://hadoop-master/classicmodels \
--username hive \
--password hive123 \
--delete-target-dir \
--table customers --m 1 \
--hive-import

19/02/01 11:55:44 INFO mapreduce.ImportJobBase: Transferred 14.0439 KB in 18.1487 seconds (792.3965 bytes/sec)
19/02/01 11:55:44 INFO mapreduce.ImportJobBase: Retrieved 122 records.
19/02/01 11:55:44 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table customers
19/02/01 11:55:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
19/02/01 11:55:44 WARN hive.TableDefWriter: Column creditLimit had to be cast to a less precise type in Hive
19/02/01 11:55:44 INFO hive.HiveImport: Loading uploaded data into Hive
19/02/01 11:55:44 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
19/02/01 11:55:44 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
... 12 more

4 REPLIES 4
Highlighted

Re: Hive and Sqoop Integration which are running on different machine

Mentor

@SAKTHIVEL G

Your sqoop command cannot locate the hive config, hence it doesn't know where the hive is running. To resolve this issue install HCat Client, Hive Client using Ambari on the slave node where you are running your sqoop commands from.

This should resolve the problem as, during the installation of these clients, the correct hive config files like hive-site.xml will be copied over.

HTH

Highlighted

Re: Hive and Sqoop Integration which are running on different machine

Mentor

@SAKTHIVEL G

Do you still need help if the issue was resolved with the provided solution please accept the answer so the thread is closed.

Highlighted

Re: Hive and Sqoop Integration which are running on different machine

Mentor

@SAKTHIVEL G

Any updates?

Highlighted

Re: Hive and Sqoop Integration which are running on different machine

Mentor

@SAKTHIVEL G

Any updates did my response resolve the issue if so accept so the thread is marked as closed.

Don't have an account?
Coming from Hortonworks? Activate your account here