Support Questions
Find answers, ask questions, and share your expertise

Kerberos between two clusters is failing

Contributor

Hi,

We have two clusters one which has all the hadoop services and the other which has just kafka and zookeeper. We have different realms for these clusters. We have enabled trust between these clusters. When i kinit with cluster A realm in cluster B and do hdfs ls of cluster A, i'm receiving below error.

hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
Warning: fs.defaultFS is not set when running "ls" command.

When i kinit with cluster B realm in cluster A and do hdfs ls of cluster A, i'm receiving below error.

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
Caused by: java.io.IOException: Couldn't setup connection for SVC_TEST@DEVKAFKA.xxxx.PRE.CORP to xxxxxxxxx.devoperational1.xxxx.pre.corp/xxxxxxx:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:710)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:416)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 41 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

... 44 more

13 REPLIES 13

Super Mentor

@Rajesh Reddy

Do you have a valid kerberos ticket from the correct Realm?

# klist 
(AND)
# klist -kte /etc/security/keytabs/YOUR_KEYTAB

.

Also please chekc the AD realm server info is correctly mentioned on your host "/etc/krb5.conf" where you are running this "hdfs" command and also the "auth_to_local" rules corresponds to your principal.

.

Contributor

Hi @Jay Kumar SenSharma

Yes i have a valid keytab and auth_to_local is configured in cluster A hdfs. cluster B does not have hdfs to configure the same.

Mentor

@Rajesh Reddy

To be able to help, can you paste here your kbr5.conf from both clusters? The entry to validate are

[realms]

[domain_realm]

Contributor

hi @Geoffrey Shelton Okot

Sorry cant paste the entire krb5. but the realm and domain realms are below.

Cluster A. [realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP Cluster B [realms] DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

Super Mentor

@Rajesh Reddy

Please run the same "hdfs" command after adding the additional DEBUGS to find the cause of failure:

# export HADOOP_ROOT_LOGGER=DEBUG,console
# export HADOOP_OPTS="-Dsun.security.krb5.debug=true ${HADOOP_OPTS}"
# export HADOOP_JAAS_DEBUG=true

# hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

.

Also please share the output of the above command.

Contributor

Contributor

Does the cluster B which has only kafka and zookeeper services need hdfs service as well to communicate with cluster A??

Mentor

@Rajesh Reddy

From the outlook, the configuration doesn't look classic Kerberos setup! In total you have 6 KDC servers, in my opinion for the 2 clusters, you should have at least 2 kdc's one for each cluster and maybe a backup KDC for each cluster that would make the total to 4.

Can you explain a bit why you have the below 6 KDC's

kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp

kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp


Cluster A

[realms] 
DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 

default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } 

[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 


Cluster B

[realms] 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp


default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 


[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

For your setup I expected to see something like this in the krb5.conf

[realms]
 DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devoperational1.xxxxxxxx.pre.corp
 }
 DEVKAFKA.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devkafka.xxxxxxxx.pre.corp
[domain_realm]
 .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP
 devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP

Could you please clarify your KDC setup's

Contributor

Hi @Geoffrey Shelton Okot

As per our organization standard, we have 3 KDC's in all the clusters.

Mentor

@Rajesh Reddy

It would be very interesting can you share the link you used to setup the KDC? Do the 3 KDC's have the same REALM is so how do they synchronise the KDC database?

How did you set up the trust between the 2 clusters? Did you follow the same procedure like below

Setting up a Realm

Trust I am wondering how you set up the trust between the 2 cluster? In my example here, the Kerberos realms are A.EXAMPLE.COM and B.EXAMPLE.COM.

You need to create the entry for the shared principal for the B realm in the A realm, using kadmin.

# kadmin -r A.EXAMPLE.COM 
kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM 
Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. 
quit 

That means that the A realm will trust the B principal. To create a bidirectional trust, then create principals going the reverse way. Create a principal for the A realm in the B realm, using kadmin.

# kadmin -r B.EXAMPLE.COM 
kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM 
Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": 
Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. 
quit 

Use the get_principal command to verify that both entries have matching key version numbers (kvno values) and encryption types.

It is also possible to reduce the number of hops and represent very complex trust flows by explicitly defining the flow. The [capaths] section of the /etc/krb5.conf file defines the trust flow between different realms.

The format of the [capaths] section is relatively straightforward: there is a main entry for each realm where a client has a principal, and then inside each realm section is a list of intermediate realms from which the client must obtain credentials.

[capaths] 
A.EXAMPLE.COM = { 
B.EXAMPLE.COM = . 
C.EXAMPLE.COM = B.EXAMPLE.COM 
D.EXAMPLE.COM = B.EXAMPLE.COM 
D.EXAMPLE.COM = C.EXAMPLE.COM } 

With credentials from Realm A, the client obtains a krbtgt/A@A ticket from the KDC of Realm A. Using this ticket, the client then asks for the krbtgt/B@A ticket.

The krbtgt/B@A ticket issued by the KDC of Realm A is a cross-realm ticket granting ticket. It allows the client to ask the KDC of Realm B for a ticket to a service principal of Realm B.

With the krbtgt/B@A ticket, the client asks for the krbtgt/C@B cross-realm ticket.

With the krbtgt/C@B ticket issued by the KDC of Realm B, the client asks for the krbtgt/D@C cross-realm ticket.

With the krbtgt/D@C ticket issued by the KDC of Realm C, the client asks the KDC of Realm D for a ticket to a service principal in Realm D.

After this, the credentials cache contains tickets for krbtgt/A@A, krbtgt/B@A, krbtgt/C@B, krbtgt/D@C, and service/hostname@D. To obtain the service/hostname@D ticket, it was required to obtain the three intermediate cross-realm tickets.

Contributor

hi @Geoffrey Shelton Okot

We are using standard MIT setup. Its just that we have 3 KDC servers nothing else defers in the configuration process. In fact i created the trust setup from the beginning and its working from Cluster A. Since Cluster B doesn't have hdfs, we cant access Cluster A hdfs with below error.

cabig02@srvbdadvlsk32:~$ hdfs dfs -ls hdfs://srvbdadvlsk21.devoperational1.xxxxxxx.pre.corp:8020/

ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Warning: fs.defaultFS is not set when running "ls" command.

cabig02@srvbdadvlsk32:~$

Rising Star

Have you kinit before running the command? The debug log showed that you are running as a non-kerberos user root.

Can you run kdestroy/kinit/klist and ensure your local ticket cache has a valid Kerberos ticket before running the hdfs command?

17/11/08 11:10:48 DEBUG security.UserGroupInformation: hadoop login commit
17/11/08 11:10:48 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: User entry: "root"
17/11/08 11:10:48 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)

Contributor

I got same error, as below

Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more Caused by: KrbException: Fail to create credential. (63) - No service creds at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

And i am able to resolve it by deleting existing krbtgt prinicipals and recreating krbtgt cross realm prinicipals on both clusters with same password. the password must be same for these prinicipals on both KDC's.


krbtgt/A.COM@B.COM

krbtgt/B.COM@A.COM


; ;