Support Questions
Find answers, ask questions, and share your expertise

Oozie - Sqoop import into hive table fail

Oozie - Sqoop import into hive table fail

New Contributor

Hello,

I have a sqoop job which imports data into a hive table

sqoop job --create job_name --meta-connect jdbc:hsqldb:hsql://hostname:16000/sqoop -- import --connect "jdbc:sqlserver://hostname;instance=1433;" --username name --password-alias pwd --incremental append --check-column col2 --last-value '2016-12-03 23:59:59.097' --table table --target-dir /tmp/folder/ --hive-import --hive-overwrite --hive-table db.table --split-by col1 --hive-drop-import-delims  -m 8

workflow.xml

<action name = "SqoopImport" >
		<sqoop xmlns = "uri:oozie:sqoop-action:0.2" >
			<job-tracker>${jobTracker}</job-tracker>
			<name-node>${nameNode}</name-node>


			<configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
			</configuration>
			<!-- mention the sqoop job under the arg parameter -->
			<arg>job</arg>
			<arg>--meta-connect</arg>
			<arg>${hsqldb}</arg>
			<arg>--exec</arg>
			<arg>${Insert_Job}</arg>	
			<file>/tmp/hive-site.xml#hive-site.xml</file>
					</sqoop>
		<ok to="hive-node"/>
        <error to="kill-email"/>
		</action>
					
			ERROR org.apache.sqoop.tool.ImportTool  - Encountered IOException running import job: java.io.IOException: Cannot run program "hive": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at java.lang.Runtime.exec(Runtime.java:620)
	at java.lang.Runtime.exec(Runtime.java:528)
	at org.apache.sqoop.util.Executor.exec(Executor.java:76)
	at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:387)
	at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:342)
	at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:246)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
	at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:243)
	at org.apache.sqoop.tool.JobTool.run(JobTool.java:298)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
	at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:202)
	at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:182)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
	at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

I have added the hive-site.xml in the workflow. but still getting error as hive not found.

Kindly let me know what I am missing here

7 REPLIES 7

Re: Oozie - Sqoop import into hive table fail

Cloudera Employee

Hi,

Make sure you having oozie sharelib updated.

hdfs dfs -ls /user/oozie/share/lib/lib_20161123162916 Found 10 items drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/distcp drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/hcatalog drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/hive drwxr-xr-x - oozie hdfs 0 2016-11-23 16:30 /user/oozie/share/lib/lib_20161123162916/hive2 drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/mapreduce-streaming drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/oozie drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/pig -rw-r--r-- 3 oozie hdfs 1367 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/sharelib.properties drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/spark drwxr-xr-x - oozie hdfs 0 2016-11-23 16:29 /user/oozie/share/lib/lib_20161123162916/sqoop

and make sure also HIVE_HOME set to in your environment!

Re: Oozie - Sqoop import into hive table fail

@nithen ac

Is your sqoop running in CLI? I'm asking to just confirm that we don't have any problem with sqoop syntax.

Re: Oozie - Sqoop import into hive table fail

New Contributor

Yes. Sqoop query is running fine

Re: Oozie - Sqoop import into hive table fail

I have this same problem.

================================================

57469 [main] DEBUG org.apache.sqoop.hive.TableDefWriter - Load statement: LOAD DATA INPATH 'hdfs://ambcluster/user/hdfs/users' INTO TABLE `users`

2016-12-19 11:19:45,530 DEBUG [main] hive.TableDefWriter (TableDefWriter.java:getLoadDataStmt(252)) - Load statement: LOAD DATA INPATH 'hdfs://ambcluster/user/hdfs/users' INTO TABLE `users`

57471 [main] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive

2016-12-19 11:19:45,532 INFO [main] hive.HiveImport (HiveImport.java:importTable(195)) - Loading uploaded data into Hive

57473 [main] DEBUG org.apache.sqoop.hive.HiveImport - Using external Hive process.

2016-12-19 11:19:45,534 DEBUG [main] hive.HiveImport (HiveImport.java:executeScript(343)) - Using external Hive process.

57477 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Cannot run program "hive": error=2, No such file or directory

================================================

When run from CLI sqoop imports fulfills without errors.

Re: Oozie - Sqoop import into hive table fail

New Contributor

Anybody got a solution for this issue. Me too facing the same issue.

Thanks,
Edwin

Re: Oozie - Sqoop import into hive table fail

Explorer

Put hive-site.xml hdfs path in <job xml>/user/..../hive-site.xml</job xml>

Re: Oozie - Sqoop import into hive table fail

Expert Contributor

Hi,

 

Can you please check if your hive gateway has been configured on all the Node manager because this sqoop command can be run from any of the NM and your hive gateway should be there.

 

Regards

Nitish