Migrating HDP cluster nodes from CentOS 6 to CentOS 7

[#1]

Starting with HDP 3.0 release, Hortonworks has stopped CentOS 6 as a supported platform. This would mean existing cluster nodes needs to be migrated to CentOS 7 before install/upgrade to HDP 3.0. However, CentOS has no official support for upgrading OS from CentOS 6 to CentOS 7. This article provides an unofficial guide for migrating cluster nodes to CentOS 7 without losing data. This is downtime migration. We would strongly recommend trying these steps on test clusters. These steps are applicable only for a smaller cluster ( <= 10 nodes ) and larger nodes we would need different strategy + automation for the same.

Below are some of the assumptions or pre-requisite to be performed before upgrading the OS.

Pre-Requisite

• Visit all service configuration for e.x. HDFS, YARN etc make sure none of the configurations are pointing to non-root disk. If the services are pointing to root disk then upgrading the OS will format the root disk and we will lose data. Please make sure to migrate all configuration pointing to root disk to non-root disk path before upgrading the OS. For .e.x. below service configs.

HDFS Data Node Directory ( dfs.datanode.data.dir) is pointing to non-root disk ( i.e. on mounted disk other than where OS is installed )
HDFS Name Node Directory (dfs.namenode.name.dir) is pointing to non-root disk
Yarn Local Directories config (yarn.nodemanager.local-dirs) is pointing to non-root disk.
Zookeeper dataDir is pointing to non-root disk.
Oozie Data Directory is pointing 

• Make sure none of the databases for services like Hive, Oozie, Ranger, and Ambari are within the cluster. If they are part of the clusters being migrated, please migrates the DBs to external hosts before upgrading the OS.
• Take a backup of cluster blueprint. Please refer to this link on exporting cluster blueprint
• Backup all required data in the cluster root disk on all nodes.

Migrating Slaves Nodes

Pick a slave node in the cluster for upgrading the OS. Please note each slave nods needs to be upgraded one at a time. Please do not attempt upgrading multiple slave node in a single go. Follow below steps to upgrade the slave node from CentOS 6 to CentOS 7.

• If the selected slave node has DataNode installed, please Decommission the DataNode. Please refer to this link for decommissioning.
• Stop all services on the node from Ambari.
• ssh to the stop ambari-agent on the node using below command.
  • ambari-agent stop

• Double check that required files on root disk is backed up. Now upgrade the OS to CentOS 7. Steps to upgrade the OS is out of scope for this article.
• Once the OS is upgraded, make sure hostname and IP address are same as before the upgrade.
• Update /etc/hosts with all the cluster nodes IP address and hostname.
• Disable firewall.
• Add the required ambari repo to /etc/yum.repos.d and install ambari agent using below command
  • yum install ambari-agent
• Install required JDK version pointing to the path configured in ambari.properties
• Reset the certificates and configure ambari-agent to connect to ambari-server using below command.
  • ambari-agent reset <ambari-server-host-name>
• ambari-agent had issues with connecting ambari-server when we were performing and below steps helped us in solving the problem.
  • Under /var/lib/ambari-server/keys on ambari-server node, move the existing csr and crt files for the upgraded host being registered.
  • Add below entry under [security] section of /etc/ambari-agent/conf/ambari-agent.ini
    • force_https_protocol=PROTOCOL_TLSv1_2

Now start the ambari-agent using below command.

• ambari-agent start

Once the host is ambari-agent is up and running, goto the Ambari and for the upgraded host run "Recover Host" option from the Host Actions.

• Recover Host reinstalls all services on the upgraded node with the required yum package.

Once the recover host is complete, recommission the DataNode.

Start all the services on the host.

Migrating Master Nodes

Upgrading the Master Nodes to CentOS 7 is similar to that of slaves node. Since this article covers upgrade with downtime, inline upgrading is out of scope. In case we are performing inline upgrade we would need to have HA for all master and move the components to different hosts before upgrading the same, however, this is out of scope from the article.

Migrating Ambari Node

Run below pre-requisite before upgrading the Ambari Node.

• Backup the key directories in ambari by running below command.
  • ambari-server backup
• Backup the ambari.properties under /etc/ambari-server/conf
• Back up the ambari-server DB if it is embedded postgres DB using below command
  • pg_dump -U {ambari.db.username} -f ambari.sql
• Preferably restore the DB on an external host by running below command
  • psql -d ambari -f /tmp/dbdumps/ambari.sql

Post the pre-requisite is complete follow below steps for the actual upgrade.

• Upgrade the OS from CentOS 6 to CentOS 7 after taking required backup on the root disk.
• Update /etc/hosts with all the cluster nodes IP address and hostname.
• Disable firewall.
• Configure ambari repo under /etc/yum.repos.d
• Install ambari-server using below command
  • yum install ambari-server
  • yum install ambari-agent
• Install required JDK version pointing to the path configured in ambari.properties
• Reset the certificates and configure ambari-agent to connect to ambari-server using below command.
  • ambari-agent reset <ambari-server-host-name>
• Run below setup command to point to the external DB. Also, set up similar to how the original ambari server is configured.
  • ambari-server setup
• Start ambari-agent using below command
  • ambari-agent start
• Start ambari-server using below command
  • ambari-server start

Migrating MIT KDC

If the cluster is configured with MIT KDC and is installed within the cluster, follow below steps to backup and restore kerberos database. Please note kdc needs to be installed on the same host where it was installed before the upgrade.

prerequisite

• Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes.
• Note down your kdc admin principal and password
• Backup /etc/krb5.conf
• Backup /var/kerberos directory Backup

backup

• Take the kerberos database dump using below command ( to be executed on the node running Kerberos )
  • kdb5_util dump kdb5_dump.txt
• Safely backup the kdb5_dump.txt.

restore

• Restore the kerberos database execute below command
  • kdb5_util load kdb5_dump.txt
• Restore the /etc/krb5.conf from backup.
• Restore /var/kerberos/krb5kdc/kdc.conf from backup.
• Restore /var/kerberos/krb5kdc/kadm5.acl from backup.
• Run below command to store master principal in stash file ( kdc admin password is required )
  • kdb5_util stash
• Start KDC server using below command
  • service krb5kdc start

[#2]

Thanks @Santhosh B Gowda. We have around 15Nodes in the cluster. The Ambari version is 2.6.1.5 & HDP 2.6.3. We are preparing the OS Upgrade plan for production, We are doing this in test region. I have the following questions a. Can you confirm if we have to upgrade Ambari2.6.1.5 to Ambari 2.7 before starting the OS Migration. This will enable us to use 'Recover Host' option introduced in the Ambari 2.7. b. Should we start with the Ambari 2.6.1.5. We will have to ensure that we have upgraded all the servers to RHEL 7 with Ambari agent & Ambari server as 2.6.1.5. Once all the Servers are upgraded to RHEL 7, We will have all the servers running Ambari 2.6.1.5 in all the servers running RHEL 7. Post this we can upgrade Ambari to 2.7. What would be your approach to this.