Starting with HDP 3.0 release, Hortonworks has stopped CentOS 6 as a supported platform. This would mean existing cluster nodes needs to be migrated to CentOS 7 before install/upgrade to HDP 3.0. However, CentOS has no official support for upgrading OS from CentOS 6 to CentOS 7. This article provides an unofficial guide for migrating cluster nodes to CentOS 7 without losing data. This is downtime migration. We would strongly recommend trying these steps on test clusters. These steps are applicable only for a smaller cluster ( <= 10 nodes ) and larger nodes we would need different strategy + automation for the same. Below are some of the assumptions or pre-requisite to be performed before upgrading the OS. Pre-Requisite Visit all service configuration for e.x. HDFS, YARN etc make sure none of the configurations are pointing to non-root disk. If the services are pointing to root disk then upgrading the OS will format the root disk and we will lose data. Please make sure to migrate all configuration pointing to root disk to non-root disk path before upgrading the OS. For .e.x. below service configs. HDFS Data Node Directory ( dfs.datanode.data.dir) is pointing to non-root disk ( i.e. on mounted disk other than where OS is installed )
HDFS Name Node Directory (dfs.namenode.name.dir) is pointing to non-root disk
Yarn Local Directories config (yarn.nodemanager.local-dirs) is pointing to non-root disk.
Zookeeper dataDir is pointing to non-root disk.
Oozie Data Directory is pointing Make sure none of the databases for services like Hive, Oozie, Ranger, and Ambari are within the cluster. If they are part of the clusters being migrated, please migrates the DBs to external hosts before upgrading the OS. Take a backup of cluster blueprint. Please refer to this link on exporting cluster blueprint Backup all required data in the cluster root disk on all nodes. Migrating Slaves Nodes Pick a slave node in the cluster for upgrading the OS. Please note each slave nods needs to be upgraded one at a time. Please do not attempt upgrading multiple slave node in a single go. Follow below steps to upgrade the slave node from CentOS 6 to CentOS 7. If the selected slave node has DataNode installed, please Decommission the DataNode. Please refer to this link for decommissioning. Stop all services on the node from Ambari. ssh to the stop ambari-agent on the node using below command. ambari-agent stop Double check that required files on root disk is backed up. Now upgrade the OS to CentOS 7. Steps to upgrade the OS is out of scope for this article. Once the OS is upgraded, make sure hostname and IP address are same as before the upgrade. Update /etc/hosts with all the cluster nodes IP address and hostname. Disable firewall. Add the required ambari repo to /etc/yum.repos.d and install ambari agent using below command yum install ambari-agent Install required JDK version pointing to the path configured in ambari.properties Reset the certificates and configure ambari-agent to connect to ambari-server using below command. ambari-agent reset <ambari-server-host-name> ambari-agent had issues with connecting ambari-server when we were performing and below steps helped us in solving the problem. Under /var/lib/ambari-server/keys on ambari-server node, move the existing csr and crt files for the upgraded host being registered. Add below entry under [security] section of /etc/ambari-agent/conf/ambari-agent.ini force_https_protocol=PROTOCOL_TLSv1_2 Now start the ambari-agent using below command. ambari-agent start Once the host is ambari-agent is up and running, goto the Ambari and for the upgraded host run "Recover Host" option from the Host Actions. Recover Host reinstalls all services on the upgraded node with the required yum package. Once the recover host is complete, recommission the DataNode. Start all the services on the host. Migrating Master Nodes
Upgrading the Master Nodes to CentOS 7 is similar to that of slaves node. Since this article covers upgrade with downtime, inline upgrading is out of scope. In case we are performing inline upgrade we would need to have HA for all master and move the components to different hosts before upgrading the same, however, this is out of scope from the article. Migrating Ambari Node Run below pre-requisite before upgrading the Ambari Node. Backup the key directories in ambari by running below command. ambari-server backup Backup the ambari.properties under /etc/ambari-server/conf Back up the ambari-server DB if it is embedded postgres DB using below command pg_dump -U {ambari.db.username} -f ambari.sql Preferably restore the DB on an external host by running below command psql -d ambari -f /tmp/dbdumps/ambari.sql Post the pre-requisite is complete follow below steps for the actual upgrade. Upgrade the OS from CentOS 6 to CentOS 7 after taking required backup on the root disk. Update /etc/hosts with all the cluster nodes IP address and hostname. Disable firewall. Configure ambari repo under /etc/yum.repos.d Install ambari-server using below command yum install ambari-server yum install ambari-agent Install required JDK version pointing to the path configured in ambari.properties Reset the certificates and configure ambari-agent to connect to ambari-server using below command. ambari-agent reset <ambari-server-host-name> Run below setup command to point to the external DB. Also, set up similar to how the original ambari server is configured. ambari-server setup Start ambari-agent using below command ambari-agent start Start ambari-server using below command ambari-server start Migrating MIT KDC If the cluster is configured with MIT KDC and is installed within the cluster, follow below steps to backup and restore kerberos database. Please note kdc needs to be installed on the same host where it was installed before the upgrade. prerequisite Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes. Note down your kdc admin principal and password Backup /etc/krb5.conf Backup /var/kerberos directory Backup backup Take the kerberos database dump using below command ( to be executed on the node running Kerberos ) kdb5_util dump kdb5_dump.txt Safely backup the kdb5_dump.txt. restore Restore the kerberos database execute below command kdb5_util load kdb5_dump.txt Restore the /etc/krb5.conf from backup. Restore /var/kerberos/krb5kdc/kdc.conf from backup. Restore /var/kerberos/krb5kdc/kadm5.acl from backup. Run below command to store master principal in stash file ( kdc admin password is required ) kdb5_util stash Start KDC server using below command service krb5kdc start
... View more
