I'm facing a new issue on our kerberized cluster :
I submit a coordinator job to oozie server, that schedules a worflow execution once a day.
Before submitting the coordinator job to oozie, I have to proceed with "kinit" command : OK, this is the expected behavior : This command returns a kerberos ticket that will be used to authenticate against oozie server when submitting my job.
The tricky part : my oozie workflow runs a java action that submits new "sub-workflows" to oozie and monitors their execution thanks to oozie java client API (org.apache.oozie.client.OozieClient.getJobInfo())
Everything has been working fine for more than 10 days, but unexpectedly it failed yesterday night, and the reason is about a kerberos ticket expiration : here is what I found in the mapper logs :
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, GSSException: No valid credentials provided (Mechanism level: Ticket expired (32)) org.apache.oozie.action.hadoop.JavaMainException: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, GSSException: No valid credentials provided (Mechanism level: Ticket expired (32)) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, GSSException: No valid credentials provided (Mechanism level: Ticket expired (32)) at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:569) at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:1038) at org.apache.oozie.client.OozieClient.getJobInfo(OozieClient.java:1016) at myjavaclass.generateWfs(WfLauncher.java:70) ...
I understood that oozie server basically submits a map/reduce job and that mappers execute my java code.
I also read that if this java action has to authenticate against hadoop cluster, it has to retrieve oozie delegation token by using "HADOOP_TOKEN_FILE_LOCATION" environment variable (populated by oozie ?)
However, it seems that "somehow", my java code is able to authenticate against oozie server to submit sub-workflows (because it has been running successfully for 10 days)...but how does it work ? the kerberos ticket is not propagated to mappers, so how is it possible ?
Above stracktrace clearly mentions "ticket expired" error, so mapper seems to use a kerberos ticket...which one ? where does it come from ?
I feel a bit confused about kerberos ticket & delegation tokens...I thought that no kerberos ticket was needed (except for the very first coordinator submission) and that everything relied on delegation token afterward...Did I miss something ?
Thanks for your help