Adding a Service RPC Port to an Existing HA Cluster with ZKFCs

[#1]

Configuring a separate service RPC port can improve the responsiveness of the NameNode by allowing DataNode and client requests to be processed via separate RPC queues.

Adding a service RPC port to an HA cluster with automatic failover via ZKFCs requires a workaround step.

The steps to configure a service RPC port are as follows:

1. Add the following settings to hdfs-site.xml.

  <property>
    <name>dfs.namenode.servicerpc-address.mycluster.nn1</name>
    <value>nn1.example.com:8040</value>
  </property>

  <property>
    <name>dfs.namenode.servicerpc-address.mycluster.nn2</name>
    <value>nn2.example.com:8040</value>
  </property>

2. Restart NameNodes.

3. Stop the ZKFC processes on both NameNodes

4. Run the following command to reset the ZKFC state in ZooKeeper

hdfs zkfc -formatZK

Without this step you will see the following exception after ZKFC restart.

java.lang.RuntimeException: Mismatched address stored in ZK for NameNode

5. Restart the ZKFCs.