What is the recommended method for changing hostnames in an Ambari managed cluster? And what other considerations should be taken in to account?

[#1]

The cluster is not Kerberized but we are migrating a cluster to a different domain along with with increased security. My questions: 1) What's the recommended method for changing the cluster hostnames and IPs so that Ambari/HDP is aware.

2) There will be new restrictions regarding iptables/ports and root access. This cluster was set up with no firewalls and password-less root access with Ambari using the root user. Aside from making sure the correct ports are open, Are there more considerations here?

[#2]

@Cassandra Spencer some checks here https://community.hortonworks.com/questions/2133/facilitating-hdp-cluster-hostname-change.html also this there's also this handy util https://github.com/u39kun/ambari-util and this blog post is pretty elaborate http://www.swiss-scalability.com/2015/01/rename-host-in-ambari-170.html

[#3]

@Cassandra Spencer Also make sure /etc/hosts reflects the new hostnames as well and /etc/sysconfig/network has a HOSTNAME entry to make it persistent across reboots.

[#4]

1) Is going to be a tricky one to handle. Ambari primarily depends on host FQDN for all operations. To change the same, the ambari db might have to be updated. Before doing any changes Ambari db backup needs to be completed. IP addresses doesn't matter. This needs to be tried on a test environment thoroughly.

2)All the port numbers required by HDP needs to be left open in iptables. If Ambari services are still going to run as root, then there would be no change.

One of the option would be to setup the second cluster afresh and use distcp to copy from existing cluster to the new one.

[#5 · Accepted solution]

Hi @Cassandra Spencer, I did this recently, and it was easier than expected: Dump the Ambari database (ambari) into an sql file, and use sed or your favorite editor to change all hostnames in the sql file. Then, run "ambari-server reset", drop ambari DB, create it again, and import the edited sql file into Postgres. Of course take all the baskups beforehand. ambarirca database is not affected. Hive DB has no references to hosts. Oozie DB has refs to old hosts but you can leave them since it's about old jobs. Hue DB, if you use Hue has refs to old hosts, and has to be updated like ambari DB. After restarting Ambari with new hostnames I had no issues. If you can keep old hostnames as aliases there is no need to change ambari agents. Otherwise you need to change ambari agent properties, to redirect agents to the new Ambari server.

Re iptables, that's unrelated to the above. You have to ensure that old required ports are opened. Starting with Ambari server and Ambari agent ports, and including all others. Also keep all ephemeral ports opened because ZooKeeper and some other components are using them.