Cluster upgrade is a well-planned maintenance activity. Before starting the upgrade, it is important to ensure that your cluster is ready for upgrade. Ambari has built-in pre-checks that are executed before the actual upgrade to check the overall health of your cluster and validate its readiness for upgrade. Let us look at the pre-checks, their purpose and how to remediate the issues, so that your cluster upgrade process is smooth. A bit of background first:
Background
Ambari supports two kinds of stack (cluster) upgrades: Rolling Upgrade (RU) and Express Upgrade (EU). Pre-checks apply to both upgrade methods, although RU has a larger set of pre-checks since the upgrade does not involve any downtime
First up, the types of pre-checks:
- Cluster level checks: Example – newer version packages should be installed on all hosts in the cluster
- Service level checks: Example – all service components should be in installed state (no failures)
- Host level checks: All hosts must be communicating with Ambari (no network issues, no heartbeat failures from Ambari agent to server)
The output of a pre-check would result in three possible values of pre-check status:
- ERROR
- WARNING
- PASS
ERRORS that have to be fixed before the upgrade can be started. WARNINGS are recommended to be fixed, but may be ignored, if you know what you are doing
When you hit ‘Upgrade’ button in Ambari User Interface (UI) – the pre-checks are kicked off in the background and within a matter of seconds you would be notified if there are some issues that need to be fixed before upgrade (RU or EU) can be started. Here is a snippet from Pre-Checks Window in Ambari UI:
Behind the scenes, below API call is triggered:
curl -u $user:$password -i -H 'X-Requested-By: ambari' -X GET http://$server:8080/api/v1/clusters/$name/rolling_upgrades_check?fields=*&UpgradeChecks/repository_v...
- $user: Ambari admin user
- $password: Password of the Ambari admin user account
- $server: Ambari server hostname or IP address
- $name: Cluster name
- $version: The actual stack version you wish to upgrade to; example: 2.5.0.0-1245 (this must be registered beforehand and package installation should be completed)
- upgrade_type: ROLLING for Rolling Upgrade, NON_ROLLING for Express Upgrade
============================================================================
Details about the pre-checks and remediation
Refer ru-eu-prechecks-details.pdf
Table above is updated as of Ambari 2.4 release
