Adding all nodes in /etc/hosts across all of them fixed the problem. Thanks! ... View more

I also have export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=zookeeper -Djava.security.auth.login.config=/usr/hdp/2.6.0.3-8/hadoop/conf/secure/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS" hdfs_jaas.conf: Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/nn.service.keytab" principal="nn/namenodehost1.local@MYREALM.FS"; }; ... View more

/etc/krb5.conf: [libdefaults] renew_lifetime = 7d forwardable = true default_realm = MYREALM.FS ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] MYREALM.FS = { admin_server = mykdc.local kdc = mykdc.local } Looking at the hadoop-hdfs-zkfc log file, I am trying to figure out where zkfc gets its zk connection string from: 2018-05-07 16:12:49,965 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server 10.169.110.22/10.169.110.22:2181. Will attempt to SASL-authenticate using Login Context section 'Client'. ... View more

HDP-2.6.0.3/ Ambari Version 2.6.1.5 Centos 7.4 (64bit) /etc/hosts file only contains FQDN entry for its host. DNS is enabled (forward ONLY). Cluster nodes: 3 zk + 2 NN (HA node) + Ranger (KMS) + 3 DN. KDC setup was done following the steps at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/install-kdc.html. All principals have been created in KDC during Kerberos enabling process. They match what was in the Excel file I'd downloaded. /var/kerberos/krb5kdc/kadm5.acl: */admin@MYREALM.FS * ... View more

I went ahead and re-built everything from scratch and still having the same issue. Any idea where ZKFC gets its ZK connection string besides ha.zookeeper.quorum ? ... View more

The provisioning script should have created it but it hadn't. Thanks! ... View more

@Rahul Pathak That didn't quite fix everything though. I was trying to put a file in an (hdfs) encryted zone and got the follwoing exception: put: java.util.concurrent.ExecutionException: org.apache.hadoop.security.authorize.AuthorizationException: User:nn not allowed to do 'GENERATE_EEK' on 'mykey'. The nn procipal should map to the 'hdfs' OS user according to the entry in hadoop.security.auth_to_local: RULE:[2:$1@$0](nn@MYREALM.COM)s/.*/hdfs/ Even after adding similar properties as above to the hdfs user hadoop.kms.proxyuser.hdfs.users=* hadoop.kms.proxyuser.hdfs.hosts=* And allowing all permissions to 'hdfs' user in the KMS policy. ... View more

That did it!. ... View more