Oozie workflow calls sqoop import (sql-server to hive) - loads data in hdfs ok but fails at hive step.

[#1]

oozie-error.txthi guys,

I am running an oozie workflow which calls sqoop import from sql-server to hive, it works ok up to data load in hdfs but fails at hive part with error "[main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Hive exited with status 1". I have tried many recommendation from blog but non is working.

I have mentioned details in attached file.

Your help will be appreciated.

Thx

[#2]

Your log has this error.

Diagnostics: File does not exist: hdfs://xxx-hdfs.abc.net:8020/user/yarn/.hiveJars/hive-exec-1.2.1.2.3.4.0-3485-bb59749376792da886f093283cc8bbdb78c69612f13abcbcedbef00717030c90.jar

Check if the /user/yarn directory exists and check that owner:group are yarn:hdfs

[#3]

Thanks Jarnes.

yes directory /user/yarn exists, however owner:group was yarn:yarn which I have changed to yarn:hdfs. After this I verified the access from user hdfs and yarn to this folder (/user/yarn/) and jar file, both users can access the directory. I ran the job again but error is still coming.

FileNotFoundException: File does not exist: hdfs://xxx-hdfs.abc.net:8020/user/yarn/.hiveJars/hive-exec-1.2.1.2.3.4.0-3485-bb59749376792da886f093283cc8bbdb78c69612f13abcbcedbef00717030c90.jar

--------------

Here is the state of folder:

[hdfs@xxx]$ hadoop fs -ls /user/yarn/.hiveJars Found 1 items -rw-r--r-- 1 yarn hdfs 20709454 2016-11-07 16:49 /user/yarn/.hiveJars/hive-exec-1.2.1.2.3.4.0-3485-bb59749376792da886f093283cc8bbdb78c69612f13abcbcedbef00717030c90.jar

--- Here I pasted the jar file from log just to make sure it exists.

[hdfs@xxx]$ hadoop fs -ls /user/yarn/.hiveJars/hive-exec-1.2.1.2.3.4.0-3485-bb59749376792da886f093283cc8bbdb78c69612f13abcbcedbef00717030c90.jar

-rw-r--r-- 1 yarn hdfs 20709454 2016-11-07 16:49 /user/yarn/.hiveJars/hive-exec-1.2.1.2.3.4.0-3485-bb59749376792da886f093283cc8bbdb78c69612f13abcbcedbef00717030c90.jar

Look like oozie is trying to access this folder other than yarn and hdfs? is this possible.

Or should I set read+write+execute for user+group for this folder.

Thx

[#4]

@Imtiaz Yousaf please set the properties given in http://stackoverflow.com/questions/24390227/hadoop-jobs-fail-when-submitted-by-users-other-than-yarn... hope this helps... Please do let me know if you still have the issue after doing the changes.

Thanks

Venkat

[#5]

@Imtiaz Yousaf

Can you please check if hive-exec-<version>.jar file is exists in your Oozie sharelib?

e.g.

[root@prodnode3 ~]# hadoop fs -ls /user/oozie/share/lib/lib_20160926083442/hive/|grep exec
-rw-r--r--   3 oozie hdfs   20755003 2016-09-26 08:34 /user/oozie/share/lib/lib_20160926083442/hive/hive-exec-1.2.1000.2.4.2.0-258.jar
[root@prodnode3 ~]#

If not the please try to re-generate oozie sharelib using below command.

#Command 1

/usr/hdp/<version>/oozie/bin/oozie-setup.sh
sharelib create -locallib /usr/hdp/<version>/oozie/oozie-sharelib.tar.gz -fs
hdfs://<active-nn>:8020

#Command 2

oozie admin -oozie http://localhost:11000/oozie
-sharelibupdate

Please run above command on oozie server as 'oozie' user.

[#6]

Thanks Venkat and Kuldeep. I gave a try to job using user "root" and it worked all well. So look like its an access related issue for other users. I will follow the steps that Venkat sent in a link those should resolve the issue. I will update the post with next finding. Thx again.

[#7 · Accepted solution]

@Imtiaz Yousaf

/Other people who are reading this.

This issue was resolved by removing hive-site.xml from below locations:

1. oozie sharelib (it was present at multiple locations in oozie sharelib)

2. from ${wf.application.path}/lib/ directory.

3. From workflow.xml (removed <job-xml> part)

By default Oozie takes this file from /etc/oozie/conf/action-conf/hive/hive-site.xml

Issue has been resolved! 🙂