question Re: Sqoop hive import through oozie in Archives of Support Questions (Read Only)

Sqoop hive import through oozie

Rarjun — Tue, 06 Jun 2017 12:01:33 GMT

I have a oozie workflow which launched Sqoop Job against Hive, While importing it is throwing the below error,

[main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50) at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:397) at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:384) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:342) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:246) at org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.main(Sqoop.java:243) at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:202) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:182) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Can someone help me out on this?

Re: Sqoop hive import through oozie

nshelke — Tue, 06 Jun 2017 12:04:14 GMT

@arjun more

You can follow below steps to get it resolved,

- Add the following classpath in file "/etc/sqoop/conf/sqoop-env.sh"

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/hdp/current/hive-client/lib/*

- Make sure you have installed the Hive client in the node where Oozie server is running.

- Add hive-exec jar in oozie lib

- Also make sure yarn user is present in HDFS(/user/yarn directory).

And restart Oozie server and try again.

Re: Sqoop hive import through oozie

Rarjun — Tue, 06 Jun 2017 12:29:21 GMT

@nshelke

I tried your suggestion, only not sure how to add user in HDFS. Can you guide me?

Re: Sqoop hive import through oozie

Rarjun — Tue, 06 Jun 2017 12:30:51 GMT

Its giving me error while running the job that, Permission denied for user yarn. Not sure why as I am running this oozie action by arjun user still getting this error.

Re: Sqoop hive import through oozie

nshelke — Tue, 06 Jun 2017 12:35:19 GMT

@arjun more

You can use below commands for adding user directory in hdfs,

su - <HDFS_USER>

hdfs dfs -mkdir /user/yarn

hdfs dfs -chown -R yarn:hdfs /user/yarn

This will create the user in HDFS.

Re: Sqoop hive import through oozie

Rarjun — Tue, 06 Jun 2017 12:38:48 GMT

@nshelke

Thanks, it really helped. And now oozie job is running.

Re: Sqoop hive import through oozie

erkansirin78 — Tue, 10 Oct 2017 20:39:40 GMT

Why add yarn user directory in hdfs?