Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberos between two clusters is failing

Highlighted

Kerberos between two clusters is failing

Contributor

Hi,

We have two clusters one which has all the hadoop services and the other which has just kafka and zookeeper. We have different realms for these clusters. We have enabled trust between these clusters. When i kinit with cluster A realm in cluster B and do hdfs ls of cluster A, i'm receiving below error.

hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
Warning: fs.defaultFS is not set when running "ls" command.

When i kinit with cluster B realm in cluster A and do hdfs ls of cluster A, i'm receiving below error.

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
Caused by: java.io.IOException: Couldn't setup connection for SVC_TEST@DEVKAFKA.xxxx.PRE.CORP to xxxxxxxxx.devoperational1.xxxx.pre.corp/xxxxxxx:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:710)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:416)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 41 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

... 44 more

13 REPLIES 13
Highlighted

Re: Kerberos between two clusters is failing

Super Mentor

@Rajesh Reddy

Do you have a valid kerberos ticket from the correct Realm?

# klist 
(AND)
# klist -kte /etc/security/keytabs/YOUR_KEYTAB

.

Also please chekc the AD realm server info is correctly mentioned on your host "/etc/krb5.conf" where you are running this "hdfs" command and also the "auth_to_local" rules corresponds to your principal.

.

Highlighted

Re: Kerberos between two clusters is failing

Contributor

Hi @Jay Kumar SenSharma

Yes i have a valid keytab and auth_to_local is configured in cluster A hdfs. cluster B does not have hdfs to configure the same.

Highlighted

Re: Kerberos between two clusters is failing

Mentor

@Rajesh Reddy

To be able to help, can you paste here your kbr5.conf from both clusters? The entry to validate are

[realms]

[domain_realm]

Highlighted

Re: Kerberos between two clusters is failing

Contributor

hi @Geoffrey Shelton Okot

Sorry cant paste the entire krb5. but the realm and domain realms are below.

Cluster A. [realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP Cluster B [realms] DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

Highlighted

Re: Kerberos between two clusters is failing

Super Mentor

@Rajesh Reddy

Please run the same "hdfs" command after adding the additional DEBUGS to find the cause of failure:

# export HADOOP_ROOT_LOGGER=DEBUG,console
# export HADOOP_OPTS="-Dsun.security.krb5.debug=true ${HADOOP_OPTS}"
# export HADOOP_JAAS_DEBUG=true

# hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/

.

Also please share the output of the above command.

Highlighted

Re: Kerberos between two clusters is failing

Contributor
Highlighted

Re: Kerberos between two clusters is failing

Contributor

Does the cluster B which has only kafka and zookeeper services need hdfs service as well to communicate with cluster A??

Highlighted

Re: Kerberos between two clusters is failing

Mentor

@Rajesh Reddy

From the outlook, the configuration doesn't look classic Kerberos setup! In total you have 6 KDC servers, in my opinion for the 2 clusters, you should have at least 2 kdc's one for each cluster and maybe a backup KDC for each cluster that would make the total to 4.

Can you explain a bit why you have the below 6 KDC's

kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp

kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp


Cluster A

[realms] 
DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 

default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } 

[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 


Cluster B

[realms] 
DEVKAFKA.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp


default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { 
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp 
kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp 
admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp 
default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } 


[domain_realm] 
.devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP 
.devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP 
devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP

For your setup I expected to see something like this in the krb5.conf

[realms]
 DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devoperational1.xxxxxxxx.pre.corp
 }
 DEVKAFKA.xxxxxxxx.PRE.CORP = {
  kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88
  admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749
  default_domain = devkafka.xxxxxxxx.pre.corp
[domain_realm]
 .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
 .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP
 devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP

Could you please clarify your KDC setup's

Highlighted

Re: Kerberos between two clusters is failing

Contributor

Hi @Geoffrey Shelton Okot

As per our organization standard, we have 3 KDC's in all the clusters.

Don't have an account?
Coming from Hortonworks? Activate your account here