Switch NameNode HA Zookeeper access from no security to SASL

[#1]

My cluster NN HA was implemented w/o securing the access to ZK. I am following the document to enable SASL: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/hdfs_co...

Here is the existing Znode used by Hadoop NN for HA. I was ablet to setACL on /hadoop-ha to sasl:nn:rwcda, but haadmin failover command failed.

[zk: zkserver1:2181(CONNECTED) 5] getAcl /hadoop-ha

'world,'anyone

: cdrwa

If I delete the old one, will NN automatically create a new Znode?

I also tried to do "hdfs zkfc -formatZK" to create a new one, but the command was not using SASL.

[#2 · Accepted solution]

@ScipioTheYounger, as described in the document you linked, you'd want to change ha.zookeeper.acl in core-site.xml to this:

<property>
    <name>ha.zookeeper.acl</name>
    <value>sasl:nn:rwcda</value>
</property>

Then, you'd want to run the following to reformat ZooKeeper for NameNode HA, which would reinitialize the znode used by NameNode HA to coordinate automatic failover.

hdfs zkfc -formatZK -force

The tricky part, as you noticed, is getting that command to authenticate with SASL. The ZooKeeper and SASL guide in the Apache documentation discusses implementation and configuration of SASL in ZooKeeper in detail. For this particular command, you can use this procedure.

First, create a JAAS configuration file at /etc/hadoop/conf/hdfs_jaas.conf:

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  useTicketCache=false
  keyTab="/etc/security/keytabs/nn.service.keytab"
  principal="nn/<HOST>@EXAMPLE.COM";
};

Note that the <HOST> will be different depending on the NameNode hostnames in your environment. Likewise, you'll need to change EXAMPLE.COM to the correct Kerberos realm.

Next, edit /etc/hadoop/conf/hadoop-env.sh, and add the following line to enable SASL when running the zkfc command.

export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=zookeeper -Djava.security.auth.login.config=/etc/hadoop/conf/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client ${HADOOP_ZKFC_OPTS}"

Then, run the "hdfs zkfc -formatZK -force" command.