Oozie job failed with an error - hive-site.xml (Permission denied)

[#1]

Here is the scenario:

1. I have workflow.xml which contains hive action.

.

2. I have added <job-xml> tag inside hive action and provided path to the hive-site.xml (say /tmp/hive-site.xml)

.

3. I have added hive-site.xml to the ${wf.application.path}/lib directory as well.

.

4. I have hive-site.xml to the oozie sharelib under:

/user/oozie/sharelib/lib_<timestamp>/oozie/hive-site.xml

/user/oozie/sharelib/lib_<timestamp>/sqoop/hive-site.xml

/user/oozie/sharelib/lib_<timestamp>/hive/hive-site.xml

.

5. My simple hive workflow is failing with below error:

Oozie Hive action configuration 
================================================================= 
Using action configuration file /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/action.xml 
------------------------ 
Setting env property for mapreduce.job.credentials.binary to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens 
------------------------ 
------------------------ 
Setting env property for tez.credentials.path to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens 
------------------------ 
<<< Invocation of Main class completed <<< 
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, hive-site.xml (Permission denied) 
java.io.FileNotFoundException: hive-site.xml (Permission denied) 
at java.io.FileOutputStream.open(Native Method) 
at java.io.FileOutputStream.<init>(FileOutputStream.java:221) 
at java.io.FileOutputStream.<init>(FileOutputStream.java:110) 
at org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite(HiveMain.java:166) 
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:196) 
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) 
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225) 
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) 
Oozie Launcher failed, finishing Hadoop job gracefully 

.

How to resolve?

.

As we know that having multiple jar files with different version in oozie sharelib can cause classnotfound exceptions, in a similar way having multiple copies of configuration file also can cause conflicts. In this case Oozie might be trying to override hive-site.xml in nodemanager's local filesystem (/hadoop/yarn/local/usercache/<username>/appcache/application_id/blah/blah with one of the copy taken from sharelib or ${wf.application.path}/lib or from the <job-xml>.

To resolve such conflicts, We have to remove extra copies of hive-site.xml from all the above mentioned location. Oozie uses hive-site.xml from /etc/oozie/conf/action-conf/hive/hive-site.xml 🙂

.

To repeat, This issue was resolved by removing hive-site.xml from below locations:

1. oozie sharelib (it was present at multiple locations in oozie sharelib)

.

2. from ${wf.application.path}/lib/ directory.

.

3. From workflow.xml (removed <job-xml> part)

.

By default Oozie takes this file from /etc/oozie/conf/action-conf/hive/hive-site.xml

.

With Oozie nothing is Easy 😉

Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! 🙂