After Kerberizing cluster, services will not start. Unable to obtain password from user

[#1]

@Chris Nelson

Check is the keytabs under /etc/security/keytabs/ are created correctly with proper hostname or not with the below command

# klist -kt /etc/security/keytabs/<keytab>

You can also try regenerating the keytabs

[#2]

@scharan

I have checked as you mentioned above. I have also tried regenerating as well. I'm still having this issue. Thank you.

[#3]

@Chris Nelson

In the first place can you share how you kerberized the cluster? The process of Kerberos database creation the contents krb5.conf.kdc.conf and kadm5.acl.

Please remember to blur sensitive data.

[#4]

@Geoffrey Shelton Okot

Thank you for your reply. I am using an existing Active Directory. I made sure of the following:

• Ambari Server and cluster hosts have network access to, and be able to resolve the DNS names of, the Domain Controllers.
• Active Directory secure LDAP (LDAPS) connectivity has been configured.
• Active Directory User container for service principals has been created and is on-hand. For example, "OU=Hadoop,OU=People,dc=apache,dc=org"
• Active Directory administrative credentials with delegated control of “Create, delete, and manage user accounts” on the previously mentioned User container are on-hand.

After following all of the documentation, I enabled Kerberos on the cluster from the Ambari dashboard. The process was able to create all of the principals in the designated container in Active Directory. I am attaching the krb5.conf and an example of the errors I receive when services attempt to start. There is no kdc.conf or kadm5.acl as I am using Active directory as my KDC.

Thank you very much.krb5conf.pngerror-example.jpeg

[#5]

@Chris Nelson

I have checked your krb5.conf and seen some inconsistencies with the REALM name and other entries I have attached at the end of this thread the correct krb5.conf file adjust it on all the nodes in the cluster and then restart the KDC and retry. Ensure you are REALM is LOCAL and NOT .LOCAL or IA.LOCAL

under [libdefaults]

default_realm = LOCAL and NOT default_realm = .LOCAL (notice the (.) dot

Under [domain_realm]

you should have 2 entries not

.local = .LOCAL

but

.local = LOCAL

local = LOCAL

under [realms]

You changed the REALM to IA.LOCAL which is NOT the same as LOCAL your actual realm so the entry should be

[realms]

LOCAL = {

The correct entry should be

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = LOCAL
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  .local = LOCAL
   local = LOCAL
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  LOCAL = {
    admin_server = xxxx.local
    kdc = xxxx.local
  }

Please revert

[#6]

@Chris Nelson

Any updates?

[#7]

@Geoffrey Shelton Okot

Thank you. I have tried the above and I am still getting the same errors. Do I need to disable Kerberos and repeat the process of configuration/enabling Kerberos with the above mentioned Realm value(s)?

Thank you very much for your assistance.