Changing dfs.nameservices value after HDFS HA has been enabled

[#1]

I recently ran into a situation where I had enabled HDFS HA and later had to change the value of dfs.nameservices. So basically during HA setup I set the value for dfs.nameservices to "MyHorton", but a couple hours later realized I should have used "MyCluster" instead.

This article explains how you can change the dfs.nameservices value after HDFS HA has been enabled already.

Background: What is the purpose of dfs.nameservices? Its the logical name of your HDFS nameservice. Its important to remember that there are several configuration parameters that have a key, which includes the actual value of dfs.nameservices, e.g. dfs.namenode.rpc-address.[nameservice id].nn1

Preparation:

• Put your HDFS in safemode and backup the namespace (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin; dfsadmin -safemode enter; dfsadmin -saveNamespace); Stop Namenode service
• Backup Hive Metastore (mysqldump hive > /tmp/mydir/backup_hive.sql)

Change Configuration:

You have to adjust the hdfs-site configuration. Change all configurations that contain the old nameservice id to the new nameservice id. In my case the new nameservice ID was "mycluster".

fs.defaultFS=hdfs:://mycluster
dfs.nameservices=mycluster
dfs.namenode.shared.edits.dir=qjournal://horton03.cloud.hortonworks.com:8485;horton02.cloud.hortonworks.com:8485;horton01.cloud.hortonworks.com:8485/mycluster
dfs.client.failover.proxy.provider.mycluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.namenode.rpc-address.mycluster.nn2=horton02.cloud.hortonworks.com:8020
dfs.ha.namenodes.mycluster=nn1,nn2
dfs.namenode.rpc-address.mycluster.nn1=horton01.cloud.hortonworks.com:8020  
dfs.namenode.http-address.mycluster.nn1=horton01.cloud.hortonworks.com:50070
dfs.namenode.http-address.mycluster.nn2=horton02.cloud.hortonworks.com:50070
dfs.namenode.https-address.mycluster.nn1=horton01.cloud.hortonworks.com:50470
dfs.namenode.https-address.mycluster.nn2=horton02.cloud.hortonworks.com:50470 

Note: You can remove the configurations that include the old nameservice id (e.g. dfs.namenode.http-address.[old_nameservice_id].nn1)

Reinit Journalnodes: This is necessary because the shared edits directory includes the nameservice id.

Please see, http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hadoop-ha/content/ha-nn-deploy-nn-cluster...

Change Hive FSRoot: It might be necessary to change the Hive metadata after the above configuration changes.

Check whether changes are necessary (as Hive-User):

hive --service metatool -listFSRoot 

If you see any table that references the old nameservice id, you have to use the following commands to switch to the new nameservice id.

Use the hive metatool to do a dry run (no actual change is made in this mode) of updating the table locations.

hive --service metatool -updateLocation hdfs://mycluster hdfs://myhorton -dryRun 

If you are satisfied with the changes the metatool will make, run the command without the -dryRun option

hive --service metatool -updateLocation hdfs://mycluster hdfs://myhorton 

Additional notes: If you are using HBase you have to adjust additional configurations.

[#2]

Additional Notes: If you are using Hive View, you have to adjust additional configurations via Ambari.