How do you enable Namenode service RPC port without HDFS service downtime?

[#1]

The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages.

Steps:

1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Property

dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port>

dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port>

dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2)

This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc..

Example from hdfs-site.xml,
dfs.nameservices=shva 
dfs.internal.nameservices=shva 
dfs.ha.namenodes.shva=nn1,nn2 
dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8020
dfs.namenode.rpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8020
dfs.namenode.handler.count=200

Service RPC host, port and handler threads:
dfs.namenode.servicerpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8040
dfs.namenode.servicerpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8040
dfs.namenode.service.handler.count=100

2) Restart Standby Namenode. You must wait till Standby Namenode out of safemode.

Note: You can check Safemode status in Standby Namenode UI.

3) Restart Active Namenode.

4) Stop Standby Namenode ZKFC controller.

5) Stop Active Namenode ZKFC controller.

6) Login Active Namenode and reset Namenode HA state.

#su - hdfs 
$hdfs zkfc -formatZK 

7) Login Standby Namenode and reset Namenode HA state.

#su - hdfs 
$hdfs zkfc -formatZK 

😎 Start Active Namenode ZKFC controller.

9) Start Standby Namenode ZKFC controller.

10) Rolling restart the Datanodes.

Note: Please check, Nodemanager should not be installed in Namenode box because it uses same port 8040. If installed then you need to change service RPC port from 8040 to different port.

Ref: scaling-the-hdfs-namenode