Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎10-08-2017

Oozie on CDH6 kerberos not getting permissions

[ Edited ]

I recently upgraded a cluster to cdh6.1.0 and everything is working apart from my jobs scheduled in oozie.  I've tried writing a new workflow and submitting it, and it is getting the same issue.  It gets a hdfs delegation token from hdfs but initially it complained that it couldn't access the file in my hdfs home directory

2019-02-12 07:18:38,582 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-2.node.hadoop.svc.cluster.local] USER[jhoran] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000001-190212060327045-oozie-oozi-W] ACTION[0000001-190212060327045-oozie-oozi-W@spark-b216] Error starting action [spark-b216]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Permission denied: user=oozie, access=EXECUTE, inode="/user/xxx":xxx:xxx:drwxrwx---

 I tried granting access explitily to oozie to see if that helped, and the script then runs.  But it goes on to complain that it can't access any keytab.

2019-02-12 06:35:18,895 [Thread-13] ERROR org.apache.thrift.transport.TSaslTransport  - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

The oozie keytab appears to be correct, I can kinit against it and access hdfs.  It's very possible that I mad a serious mistake during the upgrade process, I had to make a few changes at once such as debian-ubuntu, and changing the kerberos server to a centralized freeipa one, everything else seems to be working now though, and I'm really not sure where to even start with this one, so any help is appreciated. 

New Contributor
Posts: 4
Registered: ‎10-08-2017

Re: Oozie on CDH6 kerberos not getting permissions

[ Edited ]

I noticed that oozie could still create files using the filesystem actions in my /user/ directory, and that any files it created were still owned by my user, so clearly it was getting permissions but those permissions weren't enough to execute into directories.  So I just gave oozie execute permission on all files in my /user directory.  I'm sure this isn't the best solution, but it works for now.

 

I'm not sure what I did to resolve the second issue, but somehow that got cleaned up along the way.

 

Posts: 1,903
Kudos: 435
Solutions: 307
Registered: ‎07-31-2013

Re: Oozie on CDH6 kerberos not getting permissions

Could you attach/paste the entire exception stack trace that follows the
permission error log? Oozie should certainly be doing all its HDFS I/O
activity within an action context as the submitting user, not as its own
service user.

It would also be helpful if you can post a sample workflow configuration
that reproduces it.