Here is the scenario: 1. I have workflow.xml which contains hive action. . 2. I have added <job-xml> tag inside hive action and provided path to the hive-site.xml (say /tmp/hive-site.xml) . 3. I have added hive-site.xml to the ${wf.application.path}/lib directory as well. . 4. I have hive-site.xml to the oozie sharelib under: /user/oozie/sharelib/lib_<timestamp>/oozie/hive-site.xml /user/oozie/sharelib/lib_<timestamp>/sqoop/hive-site.xml /user/oozie/sharelib/lib_<timestamp>/hive/hive-site.xml . 5. My simple hive workflow is failing with below error: Oozie Hive action configuration
=================================================================
Using action configuration file /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/action.xml
------------------------
Setting env property for mapreduce.job.credentials.binary to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens
------------------------
------------------------
Setting env property for tez.credentials.path to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens
------------------------
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, hive-site.xml (Permission denied)
java.io.FileNotFoundException: hive-site.xml (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
at org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite(HiveMain.java:166)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:196)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Oozie Launcher failed, finishing Hadoop job gracefully . How to resolve? . As we know that having multiple jar files with different version in oozie sharelib can cause classnotfound exceptions, in a similar way having multiple copies of configuration file also can cause conflicts. In this case Oozie might be trying to override hive-site.xml in nodemanager's local filesystem (/hadoop/yarn/local/usercache/<username>/appcache/application_id/blah/blah with one of the copy taken from sharelib or ${wf.application.path}/lib or from the <job-xml>. To resolve such conflicts, We have to remove extra copies of hive-site.xml from all the above mentioned location. Oozie uses hive-site.xml from /etc/oozie/conf/action-conf/hive/hive-site.xml 🙂 . To repeat, This issue was resolved by removing hive-site.xml from below locations:
1. oozie sharelib (it was present at multiple locations in oozie sharelib)
. 2. from ${wf.application.path}/lib/ directory.
. 3. From workflow.xml (removed <job-xml> part) . By default Oozie takes this file from /etc/oozie/conf/action-conf/hive/hive-site.xml . With Oozie nothing is Easy 😉 Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! 🙂
... View more
