Using Ambari to check when upgrade / downgrade seems to be stuck

[#1]

With recent improvements in Ambari, upgrading can be done easily either using Rolling Upgrade or Express Upgrade. But there are times when upgrade/downgrade gets stuck either since all the precautions were not followed or due to product issues. When upgrade gets stuck, it is typically in an Upgrade Paused status or there aren't any status as such. At this juncture, care needs to be taken to ensure that the ambari-server is not restarted without consulting with Technical Support.

Current status of the upgrade can be checked using multiple method:

1. Ambari Log files
2. Ambari API URL's
3. Ambari Databases

1. Ambari Log files

Review Ambari Log files for any errors / exception when the upgrade is in progress.

2. Use Ambari URL's to find what are the failures

http://<ambari-server>:8080/api/v1/clusters/c1/upgrades 

For eg

http://vcert1.novalocal:8080/api/v1/clusters/VCertify/upgrades 

This would show all the upgrade or downgrade attempts. for eg if this is a upgrade failure, identify the latest Upgrade attempt number

http://vcert1.novalocal:8080/api/v1/clusters/VCertify/upgrades/119?fields=upgrade_groups/upgrade_ite... 

this would list all the actions taken as part of upgrade attempt 119.

Review the output to identify those JSON outputs without status 'COMPLETED'. This would give a clue to what items have failed to move to COMPLETED status and troubleshoot from there on.

3. Ambari Database

Note: Care has to be taken while using Ambari DB. It is mandatory to backup the database before doing any upgrade / downgrade. Following tables in the Ambari database are ideal to start troubleshooting the issues:

repo_version - this contains all the repo versions installed in the system

mysql> select repo_version_id, stack_id, version, display_name from repo_version;
+-----------------+----------+--------------+------------------+
| repo_version_id | stack_id | version      | display_name     |
+-----------------+----------+--------------+------------------+
|               1 |        4 | 2.3.0.0-2557 | HDP-2.3.0.0-2557 |
|               2 |        4 | 2.3.2.0-2950 | HDP-2.3.2.0-2950 |
|              51 |        4 | 2.3.4.0-3485 | HDP-2.3.4.0      |
+-----------------+----------+--------------+------------------+
3 rows in set (0.00 sec)

cluster_version - this contains the current versions in the cluster [installed / current / upgrading etc]

mysql> select * from cluster_version;
+----+-----------------+------------+-------------+---------------+---------------+------------+
| id | repo_version_id | cluster_id | state       | start_time    | end_time      | user_name  |
+----+-----------------+------------+-------------+---------------+---------------+------------+
|  1 |               1 |          2 | OUT_OF_SYNC | 1448369111902 | 1448369112183 | _anonymous |
|  2 |               2 |          2 | UPGRADING   | 1448521029573 | 1452063126443 | admin      |
| 51 |              51 |          2 | CURRENT     | 1450860003969 | 1451397592558 | admin      |
+----+-----------------+------------+-------------+---------------+---------------+------------+
3 rows in set (0.00 sec)

host_version - this contains the details about the versions installed in a given host

mysql> select * from host_version;
+----+-----------------+---------+-------------+
| id | repo_version_id | host_id | state       |
+----+-----------------+---------+-------------+
|  1 |               1 |       4 | OUT_OF_SYNC |
|  2 |               1 |       1 | OUT_OF_SYNC |
|  3 |               1 |       2 | OUT_OF_SYNC |
|  4 |               1 |       3 | OUT_OF_SYNC |
|  5 |               2 |       1 | UPGRADED    |
|  6 |               2 |       3 | UPGRADED    |
|  7 |               2 |       2 | UPGRADED    |
|  8 |               2 |       4 | OUT_OF_SYNC |
| 51 |              51 |       3 | CURRENT     |
| 52 |              51 |       2 | CURRENT     |
| 53 |              51 |       4 | CURRENT     |
| 54 |              51 |       1 | CURRENT     |
+----+-----------------+---------+-------------+
12 rows in set (0.05 sec)

hostcomponentstate- shows the current version / state of a given component or service

mysql> select * from hostcomponentstate;
+-----+------------+------------------------+--------------+------------------+---------------+---------+----------------+---------------+----------------+
| id  | cluster_id | component_name         | version      | current_stack_id | current_state | host_id | service_name   | upgrade_state | security_state |
+-----+------------+------------------------+--------------+------------------+---------------+---------+----------------+---------------+----------------+
|   2 |          2 | NAMENODE               | 2.3.4.0-3485 |                4 | STARTED       |       4 | HDFS           | NONE          | UNSECURED      |
|   3 |          2 | HISTORYSERVER          | 2.3.4.0-3485 |                4 | STARTED       |       1 | MAPREDUCE2     | NONE          | UNSECURED      |
|   4 |          2 | APP_TIMELINE_SERVER    | 2.3.4.0-3485 |                4 | STARTED       |       4 | YARN           | NONE          | UNSECURED      |
|   5 |          2 | RESOURCEMANAGER        | 2.3.4.0-3485 |                4 | STARTED       |       4 | YARN           | NONE          | UNSECURED      |
|   6 |          2 | WEBHCAT_SERVER         | 2.3.4.0-3485 |                4 | INSTALLED     |       2 | HIVE           | NONE          | UNSECURED      |
|   8 |          2 | HIVE_SERVER            | 2.3.4.0-3485 |                4 | INSTALLED     |       2 | HIVE           | NONE          | UNSECURED      |

Reviewing the above tables would give you an idea about the current state of upgrade / downgrade.

Further troubleshooting would depend on the current state of Ambari upgrade/ downgrade, but the above should give fair clue to troubleshoot the issues.