Created 11-08-2017 10:25 AM
Hi,
We have two clusters one which has all the hadoop services and the other which has just kafka and zookeeper. We have different realms for these clusters. We have enabled trust between these clusters. When i kinit with cluster A realm in cluster B and do hdfs ls of cluster A, i'm receiving below error.
hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/
ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
Warning: fs.defaultFS is not set when running "ls" command.
When i kinit with cluster B realm in cluster A and do hdfs ls of cluster A, i'm receiving below error.
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
Caused by: java.io.IOException: Couldn't setup connection for SVC_TEST@DEVKAFKA.xxxx.PRE.CORP to xxxxxxxxx.devoperational1.xxxx.pre.corp/xxxxxxx:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:710)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:416)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 41 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
... 44 more
Created 11-08-2017 10:30 AM
Do you have a valid kerberos ticket from the correct Realm?
# klist (AND) # klist -kte /etc/security/keytabs/YOUR_KEYTAB
.
Also please chekc the AD realm server info is correctly mentioned on your host "/etc/krb5.conf" where you are running this "hdfs" command and also the "auth_to_local" rules corresponds to your principal.
.
Created 11-08-2017 10:55 AM
Yes i have a valid keytab and auth_to_local is configured in cluster A hdfs. cluster B does not have hdfs to configure the same.
Created 11-08-2017 10:36 AM
To be able to help, can you paste here your kbr5.conf from both clusters? The entry to validate are
[realms]
[domain_realm]
Created 11-08-2017 12:02 PM
Sorry cant paste the entire krb5. but the realm and domain realms are below.
Cluster A. [realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP Cluster B [realms] DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
Created 11-08-2017 11:00 AM
Please run the same "hdfs" command after adding the additional DEBUGS to find the cause of failure:
# export HADOOP_ROOT_LOGGER=DEBUG,console # export HADOOP_OPTS="-Dsun.security.krb5.debug=true ${HADOOP_OPTS}" # export HADOOP_JAAS_DEBUG=true # hdfs dfs -ls hdfs://srvbdadvlsk20.devoperational1.xxxxxx.pre.corp:8020/
.
Also please share the output of the above command.
Created 11-08-2017 11:13 AM
Created 11-08-2017 11:42 AM
Does the cluster B which has only kafka and zookeeper services need hdfs service as well to communicate with cluster A??
Created 11-08-2017 01:16 PM
From the outlook, the configuration doesn't look classic Kerberos setup! In total you have 6 KDC servers, in my opinion for the 2 clusters, you should have at least 2 kdc's one for each cluster and maybe a backup KDC for each cluster that would make the total to 4.
Can you explain a bit why you have the below 6 KDC's
kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp
Cluster A
[realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
Cluster B
[realms] DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk37.devkafka.xxxxxxxx.pre.corp kdc = srvbdadvlsk38.devkafka.xxxxxxxx.pre.corp admin_server = srvbdadvlsk36.devkafka.xxxxxxxx.pre.corp default_domain = DEVKAFKA.xxxxxxxx.PRE.CORP } DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk21.devoperational1.xxxxxxxx.pre.corp kdc = srvbdadvlsk22.devoperational1.xxxxxxxx.pre.corp admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp default_domain = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP } [domain_realm] .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP
For your setup I expected to see something like this in the krb5.conf
[realms] DEVOPERATIONAL1.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88 admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749 default_domain = devoperational1.xxxxxxxx.pre.corp } DEVKAFKA.xxxxxxxx.PRE.CORP = { kdc = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:88 admin_server = srvbdadvlsk20.devoperational1.xxxxxxxx.pre.corp:749 default_domain = devkafka.xxxxxxxx.pre.corp [domain_realm] .devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP devoperational1.xxxxxxxx.pre.corp = DEVOPERATIONAL1.xxxxxxxx.PRE.CORP .devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP devkafka.xxxxxxxx.pre.corp = DEVKAFKA.xxxxxxxx.PRE.CORP
Could you please clarify your KDC setup's
Created 11-08-2017 02:13 PM
As per our organization standard, we have 3 KDC's in all the clusters.
Created 11-08-2017 05:01 PM
It would be very interesting can you share the link you used to setup the KDC? Do the 3 KDC's have the same REALM is so how do they synchronise the KDC database?
How did you set up the trust between the 2 clusters? Did you follow the same procedure like below
Setting up a Realm
Trust I am wondering how you set up the trust between the 2 cluster? In my example here, the Kerberos realms are A.EXAMPLE.COM and B.EXAMPLE.COM.
You need to create the entry for the shared principal for the B realm in the A realm, using kadmin.
# kadmin -r A.EXAMPLE.COM kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. quit
That means that the A realm will trust the B principal. To create a bidirectional trust, then create principals going the reverse way. Create a principal for the A realm in the B realm, using kadmin.
# kadmin -r B.EXAMPLE.COM kadmin: add_principal krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM Enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": Re-enter password for principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM": Principal "krbtgt/B.EXAMPLE.COM@A.EXAMPLE.COM" created. quit
Use the get_principal command to verify that both entries have matching key version numbers (kvno values) and encryption types.
It is also possible to reduce the number of hops and represent very complex trust flows by explicitly defining the flow. The [capaths] section of the /etc/krb5.conf file defines the trust flow between different realms.
The format of the [capaths] section is relatively straightforward: there is a main entry for each realm where a client has a principal, and then inside each realm section is a list of intermediate realms from which the client must obtain credentials.
[capaths] A.EXAMPLE.COM = { B.EXAMPLE.COM = . C.EXAMPLE.COM = B.EXAMPLE.COM D.EXAMPLE.COM = B.EXAMPLE.COM D.EXAMPLE.COM = C.EXAMPLE.COM }
With credentials from Realm A, the client obtains a krbtgt/A@A ticket from the KDC of Realm A. Using this ticket, the client then asks for the krbtgt/B@A ticket.
The krbtgt/B@A ticket issued by the KDC of Realm A is a cross-realm ticket granting ticket. It allows the client to ask the KDC of Realm B for a ticket to a service principal of Realm B.
With the krbtgt/B@A ticket, the client asks for the krbtgt/C@B cross-realm ticket.
With the krbtgt/C@B ticket issued by the KDC of Realm B, the client asks for the krbtgt/D@C cross-realm ticket.
With the krbtgt/D@C ticket issued by the KDC of Realm C, the client asks the KDC of Realm D for a ticket to a service principal in Realm D.
After this, the credentials cache contains tickets for krbtgt/A@A, krbtgt/B@A, krbtgt/C@B, krbtgt/D@C, and service/hostname@D. To obtain the service/hostname@D ticket, it was required to obtain the three intermediate cross-realm tickets.
Created 11-09-2017 09:21 AM
We are using standard MIT setup. Its just that we have 3 KDC servers nothing else defers in the configuration process. In fact i created the trust setup from the beginning and its working from Cluster A. Since Cluster B doesn't have hdfs, we cant access Cluster A hdfs with below error.
cabig02@srvbdadvlsk32:~$ hdfs dfs -ls hdfs://srvbdadvlsk21.devoperational1.xxxxxxx.pre.corp:8020/
ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
Warning: fs.defaultFS is not set when running "ls" command.
cabig02@srvbdadvlsk32:~$
Created 11-08-2017 06:39 PM
Have you kinit before running the command? The debug log showed that you are running as a non-kerberos user root.
Can you run kdestroy/kinit/klist and ensure your local ticket cache has a valid Kerberos ticket before running the hdfs command?
17/11/08 11:10:48 DEBUG security.UserGroupInformation: hadoop login commit 17/11/08 11:10:48 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 17/11/08 11:10:48 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 17/11/08 11:10:48 DEBUG security.UserGroupInformation: User entry: "root" 17/11/08 11:10:48 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)
Created 11-15-2018 10:46 AM
I got same error, as below
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more Caused by: KrbException: Fail to create credential. (63) - No service creds at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:162) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
And i am able to resolve it by deleting existing krbtgt prinicipals and recreating krbtgt cross realm prinicipals on both clusters with same password. the password must be same for these prinicipals on both KDC's.
krbtgt/A.COM@B.COM
krbtgt/B.COM@A.COM