Kerberos between two clusters is failing

[#1]

Hi,

We have two clusters one which has all the hadoop services and the other which has just kafka and zookeeper. We have different realms for these clusters. We have enabled trust between these clusters. When i kinit with cluster A realm in cluster B and do hdfs ls of cluster A, i'm receiving below error.

hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
Warning: fs.defaultFS is not set when running "ls" command.

When i kinit with cluster B realm in cluster A and do hdfs ls of cluster A, i'm receiving below error.

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
Caused by: java.io.IOException: Couldn't setup connection for SVC_TEST@DEVKAFKA.xxxx.PRE.CORP to xxxxxxxxx.devoperational1.xxxx.pre.corp/xxxxxxx:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:710)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:416)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 41 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

... 44 more

[#2]

@Rajesh Reddy

Do you have a valid kerberos ticket from the correct Realm?

# klist 
(AND)
# klist -kte /etc/security/keytabs/YOUR_KEYTAB

.

Also please chekc the AD realm server info is correctly mentioned on your host "/etc/krb5.conf" where you are running this "hdfs" command and also the "auth_to_local" rules corresponds to your principal.

.

[#3]

Hi @Jay Kumar SenSharma

Yes i have a valid keytab and auth_to_local is configured in cluster A hdfs. cluster B does not have hdfs to configure the same.

[#4]

@Rajesh Reddy

To be able to help, can you paste here your kbr5.conf from both clusters? The entry to validate are

[realms]

[domain_realm]

[#5]

hi @Geoffrey Shelton Okot

Sorry cant paste the entire krb5. but the realm and domain realms are below.

Cluster A. [realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP Cluster B [realms] DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

[#6]

@Rajesh Reddy

Please run the same "hdfs" command after adding the additional DEBUGS to find the cause of failure:

# export HADOOP_ROOT_LOGGER=DEBUG,console
# export HADOOP_OPTS="-Dsun.security.krb5.debug=true ${HADOOP_OPTS}"
# export HADOOP_JAAS_DEBUG=true

# hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

.

Also please share the output of the above command.

[#7]

cluster-b-debug-ls.txt

[#8]

Does the cluster B which has only kafka and zookeeper services need hdfs service as well to communicate with cluster A??

[#9]

@Rajesh Reddy

From the outlook, the configuration doesn't look classic Kerberos setup! In total you have 6 KDC servers, in my opinion for the 2 clusters, you should have at least 2 kdc's one for each cluster and maybe a backup KDC for each cluster that would make the total to 4.

Can you explain a bit why you have the below 6 KDC's

kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp

kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp

Cluster A

[realms] 
DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 

default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } 

[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 

Cluster B

[realms] 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp


default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 


[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

For your setup I expected to see something like this in the krb5.conf

[realms]
 DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devoperational1.xxxxxxxx.pre.corp
 }
 DEVKAFKA.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devkafka.xxxxxxxx.pre.corp
[domain_realm]
 .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP
 devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP

Could you please clarify your KDC setup's

[#10]

Hi @Geoffrey Shelton Okot

As per our organization standard, we have 3 KDC's in all the clusters.