Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberos between two clusters is failing

Re: Kerberos between two clusters is failing

Mentor

@Rajesh Reddy

It would be very interesting can you share the link you used to setup the KDC? Do the 3 KDC's have the same REALM is so how do they synchronise the KDC database?

How did you set up the trust between the 2 clusters? Did you follow the same procedure like below

Setting up a Realm

Trust I am wondering how you set up the trust between the 2 cluster? In my example here, the Kerberos realms are A.EXAMPLE.COM and B.EXAMPLE.COM.

You need to create the entry for the shared principal for the B realm in the A realm, using kadmin.

# kadmin -r A.EXAMPLE.COM 
kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM 
Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. 
quit 

That means that the A realm will trust the B principal. To create a bidirectional trust, then create principals going the reverse way. Create a principal for the A realm in the B realm, using kadmin.

# kadmin -r B.EXAMPLE.COM 
kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM 
Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. 
quit 

Use the get_principal command to verify that both entries have matching key version numbers (kvno values) and encryption types.

It is also possible to reduce the number of hops and represent very complex trust flows by explicitly defining the flow. The [capaths] section of the /etc/krb5.conf file defines the trust flow between different realms.

The format of the [capaths] section is relatively straightforward: there is a main entry for each realm where a client has a principal, and then inside each realm section is a list of intermediate realms from which the client must obtain credentials.

[capaths] 
A.EXAMPLE.COM = { 
B.EXAMPLE.COM = . 
C.EXAMPLE.COM = B.EXAMPLE.COM 
D.EXAMPLE.COM = B.EXAMPLE.COM 
D.EXAMPLE.COM = C.EXAMPLE.COM } 

With credentials from Realm A, the client obtains a krbtgt/A@A ticket from the KDC of Realm A. Using this ticket, the client then asks for the krbtgt/B@A ticket.

The krbtgt/B@A ticket issued by the KDC of Realm A is a cross-realm ticket granting ticket. It allows the client to ask the KDC of Realm B for a ticket to a service principal of Realm B.

With the krbtgt/B@A ticket, the client asks for the krbtgt/C@B cross-realm ticket.

With the krbtgt/C@B ticket issued by the KDC of Realm B, the client asks for the krbtgt/D@C cross-realm ticket.

With the krbtgt/D@C ticket issued by the KDC of Realm C, the client asks the KDC of Realm D for a ticket to a service principal in Realm D.

After this, the credentials cache contains tickets for krbtgt/A@A, krbtgt/B@A, krbtgt/C@B, krbtgt/D@C, and service/hostname@D. To obtain the service/hostname@D ticket, it was required to obtain the three intermediate cross-realm tickets.

Re: Kerberos between two clusters is failing

Contributor

hi @Geoffrey Shelton Okot

We are using standard MIT setup. Its just that we have 3 KDC servers nothing else defers in the configuration process. In fact i created the trust setup from the beginning and its working from Cluster A. Since Cluster B doesn't have hdfs, we cant access Cluster A hdfs with below error.

cabig02@srvbdadvlsk32:~$ hdfs dfs -ls hdfs://srvbdadvlsk21.devoperational1.xxxxxxx.pre.corp:8020/

ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Warning: fs.defaultFS is not set when running "ls" command.

cabig02@srvbdadvlsk32:~$

Highlighted

Re: Kerberos between two clusters is failing

Rising Star

Have you kinit before running the command? The debug log showed that you are running as a non-kerberos user root.

Can you run kdestroy/kinit/klist and ensure your local ticket cache has a valid Kerberos ticket before running the hdfs command?

17/11/08 11:10:48 DEBUG security.UserGroupInformation: hadoop login commit
17/11/08 11:10:48 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: User entry: "root"
17/11/08 11:10:48 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)

Re: Kerberos between two clusters is failing

Contributor

I got same error, as below

Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more Caused by: KrbException: Fail to create credential. (63) - No service creds at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

And i am able to resolve it by deleting existing krbtgt prinicipals and recreating krbtgt cross realm prinicipals on both clusters with same password. the password must be same for these prinicipals on both KDC's.


krbtgt/A.COM@B.COM

krbtgt/B.COM@A.COM


Don't have an account?
Coming from Hortonworks? Activate your account here