Can we automate Upgrade/Downgrade HDP

[#1]

One of our customer has questions about the rolling upgrade mechanism that they are planning to use for delivering small fixes to our systems in the field.

The context:

They are going to be delivering patches using their internal patch processes. This patch process includes the ability to ‘apply’ and also ‘remove’ patches. They were thinking of leveraging the Ambari Rolling Upgrade for the ‘apply’ mechanism for HDP distribution patches, but there may be some issues on the ‘remove’ patch mechanism.

The issue:

At the end rolling upgrade there is a decision point to ‘Finalize’, ‘Finalize Later’, or ‘Downgrade’ the upgrade. The context of the question is the web page: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_upgrading_Ambari/content/_performing_the_...

If they are going to support the removal of patches, they would need to select the ‘Finalize Later’ option for the rolling upgrade. However, the Note at the bottom of the web page indicates that the current version is the ‘starting version’, as well, they won’t be able to make configuration changes while in the ‘Finalize later’ state. This likely won’t work for them as on restarts of the Name Node and/or the other nodes in the cluster, they will need to have the patched software started by default. Also, once a patch operation is done (from thier point of view), they would need to be able to make configuration changes.

So, if they use ‘Finalize’ to finish the patch operation, according to the above web page, they would lose the ability to ‘downgrade’ the patched content back to the starting version. If there is no way to go back to the ‘starting version’, they will not be able to utilize the rolling upgrade approach.

What they are wondering is if there is a way to perform an “upgrade in reverse” after performing the ‘finalize’. Perhaps set things up to re-install the ‘starting version’ rpms and set up the respositories to utilize those pre-first-upgrade versions. Is this a viable option for them? Do we have anything better that they would be able to use?

[#2]

@rbalam

@jeff

Honestly, I would prefer some control on the upgrade process because of the complication and changes involve. It's the matter of educating and communicating to users. I don't think there is 100% automated way to upgrade/downgrade HDP as of now

[#3]

Agree! Upgrades are more complicated than just adding a service or changing some configuration. From an ops. perspective I want to see whats happening and control the upgrade process.

[#4 · Accepted solution]

@rbalam finalizing upgrade means you can't go back.

Using the finalize command removes the previous version of the NameNode and DataNode storage directories. After the upgrade is finalized, the system cannot be rolled back. Perform thorough testing of the upgraded cluster before finalizing the upgrade. The upgrade must be finalized before another upgrade can be performed.

Directories used by Hadoop 1 services set in /etc/hadoop/conf/taskcontroller.cfg are not automatically deleted after upgrade. Administrators can choose to delete these directories after the upgrade.

To finalize the upgrade, execute the following command once, on the primary NameNode host in your HDP cluster:

sudo su -l <HDFS_USER> -c "hdfs dfsadmin -finalizeUpgrade"

where <HDFS_USER> is the HDFS service user. For example, hdfs.

Here's link to the docs.