Support Questions
- Cloudera Community
- Support
- Support Questions
- Re: HDP 2.6.2 Upgrade Stuck
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
HDP 2.6.2 Upgrade Stuck
Created 10-09-2017 04:49 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Upgrading from HDP2.6.1 to HDP2.6.2 via Ambari2.5.2 failed during the finalize stage. 'Resume Upgrade' link is not available, and it appears the upgrade is stuck. Also tried 're-install', still getting the error:
Packages could not be installed
Cluster Demo1 upgrade is in progress. Cannot install packages.
Tried to manually set back to the previous version by running 'ambari-server set-current --cluster-name=Demo1 --version-display-name=HDP-2.6.1.0', and seeing the following error:
Using python /usr/bin/python
Setting current version...
Enter Ambari Admin login: dsun
Enter Ambari Admin password:
ERROR: Exiting with exit code 1.
REASON: Error during setting current version. Http status code - 500.
{
"status" : 500,
"message" : "org.apache.ambari.server.controller.spi.SystemException: Finalization failed. More details: \nSTDOUT: Begin finalizing the upgrade of cluster SCRegionDemoHDP to version 2.6.2.0-205\n\nSTDERR: The following 141 host component(s) have not been upgraded to version 2.6.2.0-205. Please install and upgrade the Stack Version on those hosts and try again.\nHost components:\nDRUID_OVERLORD on host scregionm2.field.hortonworks.com\nDRUID_SUPERSET blahblahblah.
I am assuming the upgrade process is corrupted, and bad entry on the ambari db side, any thoughts are appreciated.
Thanks.
Created 10-19-2017 09:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
During the upgrade process, a component is supposed to be restarted after the hdp-select command has been run so it will pick up the new binaries. However, the component needs to shut down and start up after the hdp-select command has been run. That way it will report to Ambari that it's version has changed and what it's current state is. In the event that you get stuck (as you did) during the upgrade you can unwind the versioning with a process like this:
- Make all pieces of the component are running
- Run `hdp-select set` command on all nodes in the cluster to set the new version. Make sure you get all of the pieces for the component (e.g. hadoop-hdfs-namenode, hadoop-ndfs-journalnode, etc.)
- Restart all processes for the component
- Verify that the O/S processes are running with the proper version of jar files
- Lather, rinse, and repeat for all components in the cluster
Once you have successfully gotten everything restarted with the proper bits, you should be able to manually finalize the upgrade with the following command to the Ambari Server:
ambari-server set-current --cluster=<custername> --version-display-name=HDP-2.6.2.0
If you get an error that components are not upgraded, you can check the components and hosts again. If everything seems ok, then you may need to tweak a table in the database. I ran into this when Atlas did not properly report the upgraded version to Ambari. NOTE: THIS SHOULD BE DONE WITH THE GUIDANCE OF HORTONWORKS SUPPORT ONLY
ambari=> SELECT h.host_name, hcs.service_name, hcs.component_name, hcs.version FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id ORDER BY hcs.version, hcs.service_name, hcs.component_name, h.host_name;
host_name | service_name | component_name | version
----------------------------------+----------------+-------------------------+-------------
scregione1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm2.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_SERVER | 2.6.1.0-129
scregionm1.field.hortonworks.com | DRUID | DRUID_BROKER | 2.6.2.0-205
scregionm1.field.hortonworks.com | DRUID | DRUID_COORDINATOR | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_OVERLORD | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_ROUTER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_SUPERSET | 2.6.2.0-205
scregione1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm0.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
. . .After verifying that you have, indeed, upgraded the components, a simple update command will set the proper version for the erroneous components and allow you to finalize the upgrade:
ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_CLIENT'; UPDATE 6 ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_SERVER'; UPDATE 1
After cycling the Ambari Server, you should be able to finalize:
[root@hostname ~]# ambari-server set-current --cluster=<cluster> --version-display-name=HDP-2.6.2.0 Using python /usr/bin/python Setting current version... Enter Ambari Admin login: <username> Enter Ambari Admin password: Current version successfully updated to HDP-2.6.2.0 Ambari Server 'set-current' completed successfully.
Created 10-09-2017 05:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @dsun,
you can see the Status of upgrade in Database,
psql ambari -U ambari
default password is bigdata
in DB execute the following command to see package versions ( to know status of upgrade)
select * from hostcomponentstate; id | cluster_id | component_name | version | current_stack_id | current_state | host_id | service_name | upgrade_state | security_state -----+------------+-------------------------+-------------+------------------+---------------+---------+----------------+---------------+---------------- 24 | 2 | HDFS_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | HDFS | NONE | UNSECURED 21 | 2 | DATANODE | 2.6.2.0-205 | 7 | STARTED | 2 | HDFS | NONE | UNSECURED 26 | 2 | HDFS_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 4 | HDFS | NONE | UNSECURED 1 | 2 | NAMENODE | 2.6.2.0-205 | 7 | STARTED | 4 | HDFS | NONE | UNSECURED 30 | 2 | MAPREDUCE2_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 4 | MAPREDUCE2 | NONE | UNSECURED 2 | 2 | SECONDARY_NAMENODE | 2.6.2.0-205 | 7 | STARTED | 1 | HDFS | NONE | UNSECURED 101 | 2 | OOZIE_SERVER | 2.6.2.0-205 | 7 | STARTED | 1 | OOZIE | NONE | UNSECURED 25 | 2 | HDFS_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | HDFS | NONE | UNSECURED 34 | 2 | TEZ_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 4 | TEZ | NONE | UNSECURED 103 | 2 | OOZIE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | OOZIE | NONE | UNSECURED 102 | 2 | OOZIE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | OOZIE | NONE | UNSECURED 52 | 2 | METRICS_MONITOR | UNKNOWN | 7 | STARTED | 1 | AMBARI_METRICS | NONE | UNSECURED 5 | 2 | HISTORYSERVER | 2.6.2.0-205 | 7 | STARTED | 1 | MAPREDUCE2 | NONE | UNSECURED 4 | 2 | APP_TIMELINE_SERVER | 2.6.2.0-205 | 7 | STARTED | 1 | YARN | NONE | UNSECURED 58 | 2 | HST_AGENT | UNKNOWN | 7 | INSTALLED | 1 | SMARTSENSE | NONE | UNSECURED 39 | 2 | HBASE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | HBASE | NONE | UNSECURED 17 | 2 | HST_SERVER | UNKNOWN | 7 | STARTED | 4 | SMARTSENSE | NONE | UNSECURED 59 | 2 | MYSQL_SERVER | UNKNOWN | 7 | STARTED | 1 | HIVE | NONE | UNSECURED 12 | 2 | ZOOKEEPER_SERVER | 2.6.2.0-205 | 7 | STARTED | 4 | ZOOKEEPER | NONE | UNSECURED 51 | 2 | METRICS_MONITOR | UNKNOWN | 7 | STARTED | 4 | AMBARI_METRICS | NONE | UNSECURED 3 | 2 | RESOURCEMANAGER | 2.6.2.0-205 | 7 | STARTED | 1 | YARN | NONE | UNSECURED 45 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | 7 | INSTALLED | 4 | AMBARI_INFRA | NONE | UNSECURED 18 | 2 | ACTIVITY_ANALYZER | UNKNOWN | 7 | INSTALLED | 4 | SMARTSENSE | NONE | UNSECURED 38 | 2 | HIVE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | HIVE | NONE | UNSECURED 54 | 2 | METRICS_MONITOR | UNKNOWN | 7 | STARTED | 2 | AMBARI_METRICS | NONE | UNSECURED 55 | 2 | HST_AGENT | UNKNOWN | 7 | INSTALLED | 4 | SMARTSENSE | NONE | UNSECURED 13 | 2 | INFRA_SOLR | UNKNOWN | 7 | STARTED | 4 | AMBARI_INFRA | NONE | UNSECURED 31 | 2 | MAPREDUCE2_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | MAPREDUCE2 | NONE | UNSECURED 11 | 2 | ZOOKEEPER_SERVER | 2.6.2.0-205 | 7 | STARTED | 3 | ZOOKEEPER | NONE | UNSECURED 14 | 2 | METRICS_GRAFANA | UNKNOWN | 7 | STARTED | 4 | AMBARI_METRICS | NONE | UNSECURED 53 | 2 | METRICS_MONITOR | UNKNOWN | 7 | STARTED | 3 | AMBARI_METRICS | NONE | UNSECURED 44 | 2 | INFRA_SOLR_CLIENT | UNKNOWN | 7 | INSTALLED | 2 | AMBARI_INFRA | NONE | UNSECURED 36 | 2 | HCAT | 2.6.2.0-205 | 7 | INSTALLED | 2 | HIVE | NONE | UNSECURED 37 | 2 | HIVE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | HIVE | NONE | UNSECURED 32 | 2 | MAPREDUCE2_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | MAPREDUCE2 | NONE | UNSECURED 20 | 2 | SPARK2_JOBHISTORYSERVER | 2.6.2.0-205 | 7 | STARTED | 4 | SPARK2 | NONE | UNSECURED 15 | 2 | METRICS_COLLECTOR | UNKNOWN | 7 | STARTED | 3 | AMBARI_METRICS | NONE | UNSECURED 23 | 2 | HBASE_REGIONSERVER | 2.6.2.0-205 | 7 | STARTED | 2 | HBASE | NONE | UNSECURED 50 | 2 | SLIDER | 2.6.2.0-205 | 7 | INSTALLED | 2 | SLIDER | NONE | UNSECURED 6 | 2 | HIVE_METASTORE | 2.6.2.0-205 | 7 | STARTED | 1 | HIVE | NONE | UNSECURED 41 | 2 | PIG | 2.6.2.0-205 | 7 | INSTALLED | 2 | PIG | NONE | UNSECURED 27 | 2 | YARN_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 4 | YARN | NONE | UNSECURED 40 | 2 | PIG | 2.6.2.0-205 | 7 | INSTALLED | 1 | PIG | NONE | UNSECURED 49 | 2 | SLIDER | 2.6.2.0-205 | 7 | INSTALLED | 1 | SLIDER | NONE | UNSECURED 46 | 2 | SPARK_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | SPARK | NONE | UNSECURED 10 | 2 | ZOOKEEPER_SERVER | 2.6.2.0-205 | 7 | STARTED | 1 | ZOOKEEPER | NONE | UNSECURED 9 | 2 | HBASE_MASTER | 2.6.2.0-205 | 7 | STARTED | 4 | HBASE | NONE | UNSECURED 48 | 2 | SPARK2_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | SPARK2 | NONE | UNSECURED 35 | 2 | TEZ_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | TEZ | NONE | UNSECURED 22 | 2 | NODEMANAGER | 2.6.2.0-205 | 7 | STARTED | 2 | YARN | NONE | UNSECURED 28 | 2 | YARN_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | YARN | NONE | UNSECURED 33 | 2 | TEZ_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | TEZ | NONE | UNSECURED 42 | 2 | ZOOKEEPER_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 1 | ZOOKEEPER | NONE | UNSECURED 43 | 2 | ZOOKEEPER_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | ZOOKEEPER | NONE | UNSECURED 47 | 2 | SPARK_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | SPARK | NONE | UNSECURED 57 | 2 | HST_AGENT | UNKNOWN | 7 | INSTALLED | 3 | SMARTSENSE | NONE | UNSECURED 29 | 2 | YARN_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 2 | YARN | NONE | UNSECURED 56 | 2 | HST_AGENT | UNKNOWN | 7 | INSTALLED | 2 | SMARTSENSE | NONE | UNSECURED 16 | 2 | ACTIVITY_EXPLORER | UNKNOWN | 7 | STARTED | 4 | SMARTSENSE | NONE | UNSECURED 104 | 2 | OOZIE_CLIENT | 2.6.2.0-205 | 7 | INSTALLED | 4 | OOZIE | NONE | UNSECURED 19 | 2 | SPARK_JOBHISTORYSERVER | 2.6.2.0-205 | 7 | STARTED | 4 | SPARK | NONE | UNSECURED 7 | 2 | WEBHCAT_SERVER | 2.6.2.0-205 | 7 | STARTED | 1 | HIVE | NONE | UNSECURED 8 | 2 | HIVE_SERVER | 2.6.2.0-205 | 7 | STARTED | 1 | HIVE | NONE | UNSECURED
select * from hostcomponentdesiredstate;
id | cluster_id | component_name | desired_stack_id | desired_state | host_id | service_name | admin_state | maintenance_state | security_state | restart_required
-----+------------+-------------------------+------------------+---------------+---------+----------------+-------------+-------------------+----------------+------------------
6 | 2 | HIVE_METASTORE | 7 | STARTED | 1 | HIVE | | OFF | UNSECURED | 0
7 | 2 | WEBHCAT_SERVER | 7 | STARTED | 1 | HIVE | | OFF | UNSECURED | 0
19 | 2 | SPARK_JOBHISTORYSERVER | 7 | STARTED | 4 | SPARK | | OFF | UNSECURED | 0
20 | 2 | SPARK2_JOBHISTORYSERVER | 7 | STARTED | 4 | SPARK2 | | OFF | UNSECURED | 0
23 | 2 | HBASE_REGIONSERVER | 7 | STARTED | 2 | HBASE | INSERVICE | OFF | UNSECURED | 0
51 | 2 | METRICS_MONITOR | 7 | STARTED | 4 | AMBARI_METRICS | INSERVICE | OFF | UNSECURED | 0
52 | 2 | METRICS_MONITOR | 7 | STARTED | 1 | AMBARI_METRICS | INSERVICE | OFF | UNSECURED | 0
53 | 2 | METRICS_MONITOR | 7 | STARTED | 3 | AMBARI_METRICS | INSERVICE | OFF | UNSECURED | 0
54 | 2 | METRICS_MONITOR | 7 | STARTED | 2 | AMBARI_METRICS | INSERVICE | OFF | UNSECURED | 0
55 | 2 | HST_AGENT | 7 | STARTED | 4 | SMARTSENSE | INSERVICE | OFF | UNSECURED | 0
56 | 2 | HST_AGENT | 7 | STARTED | 2 | SMARTSENSE | INSERVICE | OFF | UNSECURED | 0
57 | 2 | HST_AGENT | 7 | STARTED | 3 | SMARTSENSE | INSERVICE | OFF | UNSECURED | 0
58 | 2 | HST_AGENT | 7 | STARTED | 1 | SMARTSENSE | INSERVICE | OFF | UNSECURED | 0
59 | 2 | MYSQL_SERVER | 7 | STARTED | 1 | HIVE | | OFF | UNSECURED | 0
102 | 2 | OOZIE_CLIENT | 7 | INSTALLED | 1 | OOZIE | | OFF | UNSECURED | 0
104 | 2 | OOZIE_CLIENT | 7 | INSTALLED | 4 | OOZIE | | OFF | UNSECURED | 0
1 | 2 | NAMENODE | 7 | STARTED | 4 | HDFS | | OFF | UNSECURED | 0
2 | 2 | SECONDARY_NAMENODE | 7 | STARTED | 1 | HDFS | | OFF | UNSECURED | 0
3 | 2 | RESOURCEMANAGER | 7 | STARTED | 1 | YARN | | OFF | UNSECURED | 0
4 | 2 | APP_TIMELINE_SERVER | 7 | STARTED | 1 | YARN | | OFF | UNSECURED | 0
5 | 2 | HISTORYSERVER | 7 | STARTED | 1 | MAPREDUCE2 | | OFF | UNSECURED | 0
21 | 2 | DATANODE | 7 | STARTED | 2 | HDFS | INSERVICE | OFF | UNSECURED | 0
22 | 2 | NODEMANAGER | 7 | STARTED | 2 | YARN | INSERVICE | OFF | UNSECURED | 0
24 | 2 | HDFS_CLIENT | 7 | INSTALLED | 1 | HDFS | | OFF | UNSECURED | 0
25 | 2 | HDFS_CLIENT | 7 | INSTALLED | 2 | HDFS | | OFF | UNSECURED | 0
26 | 2 | HDFS_CLIENT | 7 | INSTALLED | 4 | HDFS | | OFF | UNSECURED | 0
27 | 2 | YARN_CLIENT | 7 | INSTALLED | 4 | YARN | | OFF | UNSECURED | 0
28 | 2 | YARN_CLIENT | 7 | INSTALLED | 1 | YARN | | OFF | UNSECURED | 0
29 | 2 | YARN_CLIENT | 7 | INSTALLED | 2 | YARN | | OFF | UNSECURED | 0
30 | 2 | MAPREDUCE2_CLIENT | 7 | INSTALLED | 4 | MAPREDUCE2 | | OFF | UNSECURED | 0
31 | 2 | MAPREDUCE2_CLIENT | 7 | INSTALLED | 1 | MAPREDUCE2 | | OFF | UNSECURED | 0
32 | 2 | MAPREDUCE2_CLIENT | 7 | INSTALLED | 2 | MAPREDUCE2 | | OFF | UNSECURED | 0
33 | 2 | TEZ_CLIENT | 7 | INSTALLED | 2 | TEZ | | OFF | UNSECURED | 0
34 | 2 | TEZ_CLIENT | 7 | INSTALLED | 4 | TEZ | | OFF | UNSECURED | 0
35 | 2 | TEZ_CLIENT | 7 | INSTALLED | 1 | TEZ | | OFF | UNSECURED | 0
36 | 2 | HCAT | 7 | INSTALLED | 2 | HIVE | | OFF | UNSECURED | 0
37 | 2 | HIVE_CLIENT | 7 | INSTALLED | 2 | HIVE | | OFF | UNSECURED | 0
38 | 2 | HIVE_CLIENT | 7 | INSTALLED | 1 | HIVE | | OFF | UNSECURED | 0
39 | 2 | HBASE_CLIENT | 7 | INSTALLED | 2 | HBASE | | OFF | UNSECURED | 0
40 | 2 | PIG | 7 | INSTALLED | 1 | PIG
| | OFF | UNSECURED | 0
41 | 2 | PIG | 7 | INSTALLED | 2 | PIG | | OFF | UNSECURED | 0
42 | 2 | ZOOKEEPER_CLIENT | 7 | INSTALLED | 1 | ZOOKEEPER | | OFF | UNSECURED | 0
43 | 2 | ZOOKEEPER_CLIENT | 7 | INSTALLED | 2 | ZOOKEEPER | | OFF | UNSECURED | 0
44 | 2 | INFRA_SOLR_CLIENT | 7 | INSTALLED | 2 | AMBARI_INFRA | | OFF | UNSECURED | 0
45 | 2 | INFRA_SOLR_CLIENT | 7 | INSTALLED | 4 | AMBARI_INFRA | | OFF | UNSECURED | 0
46 | 2 | SPARK_CLIENT | 7 | INSTALLED | 1 | SPARK | | OFF | UNSECURED | 0
47 | 2 | SPARK_CLIENT | 7 | INSTALLED | 2 | SPARK | | OFF | UNSECURED | 0
48 | 2 | SPARK2_CLIENT | 7 | INSTALLED | 2 | SPARK2 | | OFF | UNSECURED | 0
49 | 2 | SLIDER | 7 | INSTALLED | 1 | SLIDER | | OFF | UNSECURED | 0
50 | 2 | SLIDER | 7 | INSTALLED | 2 | SLIDER | | OFF | UNSECURED | 0
8 | 2 | HIVE_SERVER | 7 | STARTED | 1 | HIVE | | OFF | UNSECURED | 0
9 | 2 | HBASE_MASTER | 7 | STARTED | 4 | HBASE | | OFF | UNSECURED | 0
10 | 2 | ZOOKEEPER_SERVER | 7 | STARTED | 1 | ZOOKEEPER | | OFF | UNSECURED | 0
11 | 2 | ZOOKEEPER_SERVER | 7 | STARTED | 3 | ZOOKEEPER | | OFF | UNSECURED | 0
12 | 2 | ZOOKEEPER_SERVER | 7 | STARTED | 4 | ZOOKEEPER | | OFF | UNSECURED | 0
13 | 2 | INFRA_SOLR | 7 | STARTED | 4 | AMBARI_INFRA | | OFF | UNSECURED | 0
14 | 2 | METRICS_GRAFANA | 7 | STARTED | 4 | AMBARI_METRICS | | OFF | UNSECURED | 0
15 | 2 | METRICS_COLLECTOR | 7 | STARTED | 3 | AMBARI_METRICS | | OFF | UNSECURED | 0
16 | 2 | ACTIVITY_EXPLORER | 7 | STARTED | 4 | SMARTSENSE | | OFF | UNSECURED | 0
17 | 2 | HST_SERVER | 7 | STARTED | 4 | SMARTSENSE | | OFF | UNSECURED | 0
18 | 2 | ACTIVITY_ANALYZER | 7 | STARTED | 4 | SMARTSENSE | | OFF | UNSECURED | 0
103 | 2 | OOZIE_CLIENT | 7 | INSTALLED | 2 | OOZIE | | OFF | UNSECURED | 0
101 | 2 | OOZIE_SERVER | 7 | STARTED | 1 | OOZIE | | OFF | UNSECURED | 0
select * from host_version; id | repo_version_id | host_id | state ----+-----------------+---------+--------- 2 | 1 | 3 | CURRENT 3 | 1 | 4 | CURRENT 4 | 1 | 1 | CURRENT 1 | 1 | 2 | CURRENT
select * from cluster_version; id | repo_version_id | cluster_id | state | start_time | end_time | user_name ----+-----------------+------------+---------+---------------+---------------+----------- 1 | 1 | 2 | CURRENT | 1505793403390 | 1505794154404 | admin
you can update the host_versions to UPGRADING incase some upgrades has not happened.
you can troubleshoot more based on output of this database commands.
Created 10-10-2017 02:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Akhil S Naik updating the state column from either 'CURRENT' or 'INSTALLED' to 'UPGRADING' is causing Ambari fails to start for me:
ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.
Created 10-09-2017 05:59 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please try the following:
1). For the Missing Upgrade button issue, Please try the following URL to open ambari in Experimental mode and check the "http://$AMBARI_HOST:8080/#/experimental" and save it in the UI to see if it helps:
opsDuringRollingUpgrade = true (Checked)
2). Regarding the DRUID upgrade issue. Please login to the host "scregionm2.field.hortonworks.com" via SSH and then check the following:
a). If the Druid package is upgraded?
b). Check if all the verison are reflecting "2.6.2.0-205" version, in the output of the hdp-select output. If all installed components are correctly showing "2.6.2.0-205" (except DRUID) then refere to Step c.
# hdp-select
c). If you have installed DRUID from some third party sources then run the following command to set it's version to 2.6.2.0-205
Syntax:
# hdp-select set <PACKAGE_NAME> 2.6.2.0-205
Here PACKAGE_NAME will be druid name.
(OR)
# hdp-select set all 2.6.2.0-205
.
Created on 10-10-2017 02:01 AM - edited 08-17-2019 08:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Checked "opsDuringRollingUpgrade =true", and still not seeing the resume upgrade button. Just to make myself clear, I can see the upgrade button, as shown. However, when I click it, and select express upgrade, it won't pass the check, and saying there is already an upgrade in process, but wouldn't let me resume the previous upgrade.
Created 10-10-2017 03:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please do not change any table state to "UPGRADING" that will cause issues.
As you are getting the following message:
Reason: There is an existing upgrade from 2.6.1.0-129 to 2.6.2.0-205 which has not completed. This upgrade must be completed before a new upgrade or downgrade can begin.
It will be best if you can share the output of the following tables queries which seems to be not alright:
ambari=> SELECT * FROM host_version; ambari=> SELECT * FROM cluster_version; ambari=> SELECT * FROM clusterstate; ambari=> SELECT repo_version_id, stack_id, version FROm repo_version;
.
Can you please share the output of the above SQL statements?
Created on 10-10-2017 02:39 PM - edited 08-17-2019 08:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @Jay SenSharma. Please see the output below:
Created 10-19-2017 09:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
During the upgrade process, a component is supposed to be restarted after the hdp-select command has been run so it will pick up the new binaries. However, the component needs to shut down and start up after the hdp-select command has been run. That way it will report to Ambari that it's version has changed and what it's current state is. In the event that you get stuck (as you did) during the upgrade you can unwind the versioning with a process like this:
- Make all pieces of the component are running
- Run `hdp-select set` command on all nodes in the cluster to set the new version. Make sure you get all of the pieces for the component (e.g. hadoop-hdfs-namenode, hadoop-ndfs-journalnode, etc.)
- Restart all processes for the component
- Verify that the O/S processes are running with the proper version of jar files
- Lather, rinse, and repeat for all components in the cluster
Once you have successfully gotten everything restarted with the proper bits, you should be able to manually finalize the upgrade with the following command to the Ambari Server:
ambari-server set-current --cluster=<custername> --version-display-name=HDP-2.6.2.0
If you get an error that components are not upgraded, you can check the components and hosts again. If everything seems ok, then you may need to tweak a table in the database. I ran into this when Atlas did not properly report the upgraded version to Ambari. NOTE: THIS SHOULD BE DONE WITH THE GUIDANCE OF HORTONWORKS SUPPORT ONLY
ambari=> SELECT h.host_name, hcs.service_name, hcs.component_name, hcs.version FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id ORDER BY hcs.version, hcs.service_name, hcs.component_name, h.host_name;
host_name | service_name | component_name | version
----------------------------------+----------------+-------------------------+-------------
scregione1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm2.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw0.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionw1.field.hortonworks.com | ATLAS | ATLAS_CLIENT | 2.6.1.0-129
scregionm0.field.hortonworks.com | ATLAS | ATLAS_SERVER | 2.6.1.0-129
scregionm1.field.hortonworks.com | DRUID | DRUID_BROKER | 2.6.2.0-205
scregionm1.field.hortonworks.com | DRUID | DRUID_COORDINATOR | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_HISTORICAL | 2.6.2.0-205
scregionw0.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionw1.field.hortonworks.com | DRUID | DRUID_MIDDLEMANAGER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_OVERLORD | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_ROUTER | 2.6.2.0-205
scregionm2.field.hortonworks.com | DRUID | DRUID_SUPERSET | 2.6.2.0-205
scregione1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm0.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
scregionm1.field.hortonworks.com | HBASE | HBASE_CLIENT | 2.6.2.0-205
. . .After verifying that you have, indeed, upgraded the components, a simple update command will set the proper version for the erroneous components and allow you to finalize the upgrade:
ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_CLIENT'; UPDATE 6 ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_SERVER'; UPDATE 1
After cycling the Ambari Server, you should be able to finalize:
[root@hostname ~]# ambari-server set-current --cluster=<cluster> --version-display-name=HDP-2.6.2.0 Using python /usr/bin/python Setting current version... Enter Ambari Admin login: <username> Enter Ambari Admin password: Current version successfully updated to HDP-2.6.2.0 Ambari Server 'set-current' completed successfully.
