Kerberos Cache in IPA /RedHat IDM (KEYRING) SOLVED!!

[#1]

Hi Everyone,

I came across a Kerberos cache issue and wanted to share and possible have more ideas.

One of our clients have RedHat IDM (supported version of freeIpa) and when you install sssd along with krb5 by IDM the default cache setting is 'KEYRING' than 'File'

You will still be able to get the tickets but will have GSSException error on Cluster.

KEYRING persistent cache setting works with many applications however not with HADOOP Cluster, I'm sure HDO engineering team must be looking into this for a solution, since KEYRING is the future for kerberos cache and the stuff you can do with it like keylist and etc.

To solve the errors you can comment out the "default_ccache_name=KEYRING....." on krb.conf or change it to "default_ccache_name = FILE:/tmp/krb5cc_%{uid}"

Logout and log in again - destroy the previous tickets and you should have something like "Ticket cache: FILE:/tmp/krb5cc_" in your klist output.

If you still see KEYRING PERSISTENT, kill all the running sessions of the user having the problem and restart SSSD service.

Thanks

[#2]

@mkataria perhaps you would consider posting this as an article instead of a question? It would be a useful article because others have run into this in the past (see here)

[#3]

Thanks @Ali Bajwa, posted as article

[#4 · Accepted solution]

@mkataria, thanks for this

"I'm sure HDO engineering team must be looking into this for a solution, since KEYRING is the future for kerberos cache and the stuff you can do with it like keylist and etc."

Hadoop uses the Kerberos libraries that come with the JVM. That's pretty problematic, because we have to delve into the private com.sun. and com.ibm. internal classes to get some stuff done, and that tends to break across major (and sometimes minor) releases. If we stay with the Oracle libraries, we're stuck with whatever they implement, which, as you've noticed, is pretty limited and out of date.

You may be interested to hear of Apache Kerby, an ongoing project to have a pure Java Kerberos Domain Controller and, importantly for production Hadoop —a modern Kerberos client library, with CPU-acceleration of encryption when available (i.e. latest intel parts, as Hadoop already does for HDFS encryption), and broader protocol support. I don't think its ready for production use yet, and nobody has —yet— gone beyond looking at what it would take to switch Hadoop over to it. But be assured: people are looking at it.

Finally, what was the error message you got? Was it the classic "unable to find TGT for user" kind of text, or something new? I'm trying to build a list list of common Kerberos error messages —adding a new message or a new possible cause of an existing message is something else to put in here.

[#5]

@stevel thanks for the explanation upcoming Kerby, we had the same error message which you have already documented however a different cause 'may be unsupported cache type" the error was

No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt

we commented out "default_ccache_name = KEYRING:persistent:%{uid}" and it fixed.

Thanks Mayank

[#6]

Looks like that freeipa-client on CentOS6.x is not adding the default_ccache_name entry in the /etc/krb5.conf while it adds it in CentOS7, and the KEYRING: thing makes it all go bonkers indeed.

Thanks a lot for the pointer!!!!

[#7]

THANK YOU! I barely can imagine how many nights you wasted on this. This vicious issue appeared suddenly when we installed FreeIPA client on centos 7 nodes. Whereas, on CentOS 6.7, the KEYRING was absent...

[#8]

Please also note the exact same error can be caused by missing Oracle JCE jar files. It turns out openJDK ships JCE policy jar files compatible with HDP, but not the Oracle JVM. Hence, the extensions should be installed (even if the $JAVA_HOME/lib/security folder already contain the jars, they may not be then proper ones). I learnt this the hard way.

[#9]

Using KEYRING was a state of art at the moment Kerberos was bundled for RHEL7. However moving forward into the world of containers using KEYRING becomes a challenge thus SSSD is building internal ticket cache that will be supported by the system Kerberos libraries. So in general the recommendation nowadays is to use native OS Kerberos libraries, they are most recent and will provide latest functionality and experience.