question Re: Cluster non operational after enabling Kerberos. in Support Questions

Cluster non operational after enabling Kerberos.

sadek_mostefai — Fri, 16 Sep 2022 13:10:19 GMT

Hi,

I had enabled Kerberos on my cluster w/o realizing that the hostname was never included on /etc/hosts. I went and did that and also remove and re-add Kerberos. I still cannot get rid of this error:

nn/namenodehost1.local@MYREALM.FS for zookeeper/10.169.110.22@MYREALM.FS, Server not found in Kerberos database

As if the _HOST var doesn't get translated to the host's FQDN.

Any help is really appreciated.

Sadek

Re: Cluster non operational after enabling Kerberos.

sadek_mostefai — Tue, 08 May 2018 18:53:16 GMT

I went ahead and re-built everything from scratch and still having the same issue. Any idea where ZKFC gets its ZK connection string besides ha.zookeeper.quorum ?

Re: Cluster non operational after enabling Kerberos.

Shelton — Tue, 08 May 2018 19:42:40 GMT

@Sadek M

I would gladly help, but I would need you to share all the steps you executed and the below info.

HDP /Ambari versions
Cluster OS
/etc/hosts entry [Assuming they are identical]
Number of nodes in Cluster
KDC setup process
Ambari Kerberos enabling errors.
Output of

# kadmin.local           
listprincs

kadm5.acl usually in /var/kerberos/krb5kdc
krb5.conf in /etc

Please obfuscate any hostname or sensitive info before sharing

Re: Cluster non operational after enabling Kerberos.

sadek_mostefai — Tue, 08 May 2018 20:19:33 GMT

HDP-2.6.0.3/ Ambari Version 2.6.1.5
Centos 7.4 (64bit)
/etc/hosts file only contains FQDN entry for its host. DNS is enabled (forward ONLY).
Cluster nodes: 3 zk + 2 NN (HA node) + Ranger (KMS) + 3 DN.
KDC setup was done following the steps at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/install-kdc.html. All principals have been created in KDC during Kerberos enabling process. They match what was in the Excel file I'd downloaded.

/var/kerberos/krb5kdc/kadm5.acl:
*/admin@MYREALM.FS    *

Re: Cluster non operational after enabling Kerberos.

sadek_mostefai — Tue, 08 May 2018 20:23:25 GMT

/etc/krb5.conf:


[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = MYREALM.FS
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
 
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
 
[realms]
  MYREALM.FS = {
    admin_server = mykdc.local
    kdc = mykdc.local
  }

Looking at the hadoop-hdfs-zkfc log file, I am trying to figure out where zkfc gets its zk connection string from:

2018-05-07 16:12:49,965 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server 10.169.110.22/10.169.110.22:2181. Will attempt to SASL-authenticate using Login Context section 'Client'.

Re: Cluster non operational after enabling Kerberos.

sadek_mostefai — Tue, 08 May 2018 20:29:01 GMT

I also have

 export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true 
 -Dzookeeper.sasl.client.username=zookeeper 
 -Djava.security.auth.login.config=/usr/hdp/2.6.0.3-8/hadoop/conf/secure/hdfs_jaas.conf 
 -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS"

hdfs_jaas.conf:

Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      useTicketCache=false
      keyTab="/etc/security/keytabs/nn.service.keytab"
      principal="nn/namenodehost1.local@MYREALM.FS";
};

Re: Cluster non operational after enabling Kerberos.

Shelton — Tue, 08 May 2018 20:51:10 GMT

@Sadek M

A properly functioning DNS server for your domain and functioning DNS resolvers on machines participating in your Kerberos realm is essential for the proper operation of your realm.

Kerberos can use DNS as a service location protocol, by using the DNS SRV record as defined in RFC 2052 or use a TXT record to locate the appropriate realm for a given host or domain name.

Are you using a MIT Kerberos? Can you update your krb5.conf on all the nodes by adding:

[libdefaults] 
    rdns = false

Your problem is a DNS issue, that's the reason I wanted the entries in /etc/hosts. A workaround if you cluster is small you could propagate the correct hosts' files, while you resolve the DNS issue.

Setting Up KDC Discovery Over DNS

To use KDC discovery over DNS, the following records should be placed in the zone file corresponding to the Kerberos realm. In most cases, since the Kerberos realm name is simply an uppercase version of the DNS domain owned by the organization, these DNS entries are placed into the organization’s existing DNS zone file.

Note, however, if the Kerberos realm and DNS domain differ, then a new zone must be created with the name of the Kerberos realm typical your network team should be able to help with the DNS zone update !

Your zone file example

_kerberos._udp.MYREALM.FS.     IN SRV 10 0 88  {your_kdc_server}.myrealm.fs.
_kerberos._tcp.MYREALM.FS.     IN SRV 10 0 88  {your_kdc_server}.myrealm.fs.
_kerberos-adm._tcp.MYREALM.FS. IN SRV 1  0 749 {your_kdc_server}.myrealm.fs.

Hope that helps

Re: Cluster non operational after enabling Kerberos.

Shelton — Wed, 09 May 2018 04:39:33 GMT

@Sadek M

I think you also forgot the entry [domain_realm] I have added it to your original krb5.conf, please backup your current krb5.conf and just copy and paste the one below,

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = MYREALM.FS
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[domain_realm]
  .myrealm.fs = MYREALM.FS
  myrealm.fs = MYREALM.FS

[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log

[realms]
  MYREALM.FS = {
    admin_server = mykdc.local
    kdc = mykdc.local
  }

Then restart below Kerberos daemons

# service krb5kdc start 
# service kadmin restart

Please let me know

Re: Cluster non operational after enabling Kerberos.

sadek_mostefai — Wed, 09 May 2018 04:55:21 GMT

Adding all nodes in /etc/hosts across all of them fixed the problem. Thanks!