security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before

[#1]

Hi Team,

I have integerated my kerberos cluster to AD, but when I am executing hadoop command then getting following error.

security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before

17/03/12 06:32:35 WARN ipc.Client: Couldn't setup connection for sonu@AD.COM to m1.hdp22/192.168.56.41:8020

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)

at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

[#2]

I have following value in my krb5.conf

[libdefaults]

renew_lifetime = 7d

forwardable = true

default_realm = HADOOPADMIN.COM

ticket_lifetime = 24h

dns_lookup_realm = false

dns_lookup_kdc = false

#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[domain_realm]

m1.hdp22 = HADOOPADMIN.COM

adserver.ad.com = AD.COM

[logging]

default = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

kdc = FILE:/var/log/krb5kdc.log

[realms]

HADOOPADMIN.COM = {

admin_server = m1.hdp22

kdc = m1.hdp22

}

AD.COM = {

kdc = adserver.ad.com:88

master_kdc = adserver.ad.com:88

kpasswd = adserver.ad.com:464

kpasswd_server = adserver.ad.com:464

}

[#3]

Any help ?

[#4]

Hello @Saurabh have you configured a one way trust relationship between the kdc and the AD?

If you didn't please check https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/kerb-oneway-trust.html

[#5]

Hello @Saurabh,

If you look the error message closely, it says 'No service creds'. Since you are running hadoop command, this most probably means that the NameNode service keytab is either missing or not good. For both the cases, please check NameNode log for any error during service startup.

To verify the service keytabs, try running these on NameNode:

su - hdfs
kinit -kt /etc/security/keytabs/nn.service.keytab nn/<nn-host-fqdn>@REALM

The last command should give you a correct TGT for NN service principal, that would show that NN service keytab is good.

Lastly, you can try to regenerate the keytabs for all the services.

Hope this helps !