Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

HDP 2.6.2 Upgrade Stuck

avatar
author-rank dsun author-rank
Expert Contributor

Created ‎10-09-2017 04:49 AM

Upgrading from HDP2.6.1 to HDP2.6.2 via Ambari2.5.2 failed during the finalize stage. 'Resume Upgrade' link is not available, and it appears the upgrade is stuck. Also tried 're-install', still getting the error:


Packages could not be installed

Cluster Demo1 upgrade is in progress. Cannot install packages.

Tried to manually set back to the previous version by running 'ambari-server set-current --cluster-name=Demo1 --version-display-name=HDP-2.6.1.0', and seeing the following error:

Using python /usr/bin/python Setting current version... Enter Ambari Admin login: dsun Enter Ambari Admin password: ERROR: Exiting with exit code 1. REASON: Error during setting current version. Http status code - 500. { "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: Finalization failed. More details: \nSTDOUT: Begin finalizing the upgrade of cluster SCRegionDemoHDP to version 2.6.2.0-205\n\nSTDERR: The following 141 host component(s) have not been upgraded to version 2.6.2.0-205. Please install and upgrade the Stack Version on those hosts and try again.\nHost components:\nDRUID_OVERLORD on host scregionm2.field.hortonworks.com\nDRUID_SUPERSET blahblahblah.

I am assuming the upgrade process is corrupted, and bad entry on the ambari db side, any thoughts are appreciated.

Thanks.

6,084 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank emaxwell author-rank
Guru

Created ‎10-19-2017 09:11 PM

@dsun

During the upgrade process, a component is supposed to be restarted after the hdp-select command has been run so it will pick up the new binaries. However, the component needs to shut down and start up after the hdp-select command has been run. That way it will report to Ambari that it's version has changed and what it's current state is. In the event that you get stuck (as you did) during the upgrade you can unwind the versioning with a process like this:

  1. Make all pieces of the component are running
  2. Run `hdp-select set` command on all nodes in the cluster to set the new version. Make sure you get all of the pieces for the component (e.g. hadoop-hdfs-namenode, hadoop-ndfs-journalnode, etc.)
  3. Restart all processes for the component
  4. Verify that the O/S processes are running with the proper version of jar files
  5. Lather, rinse, and repeat for all components in the cluster

Once you have successfully gotten everything restarted with the proper bits, you should be able to manually finalize the upgrade with the following command to the Ambari Server:

ambari-server set-current --cluster=<custername> --version-display-name=HDP-2.6.2.0

If you get an error that components are not upgraded, you can check the components and hosts again. If everything seems ok, then you may need to tweak a table in the database. I ran into this when Atlas did not properly report the upgraded version to Ambari. NOTE: THIS SHOULD BE DONE WITH THE GUIDANCE OF HORTONWORKS SUPPORT ONLY

ambari=> SELECT h.host_name, hcs.service_name, hcs.component_name, hcs.version FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id ORDER BY hcs.version, hcs.service_name, hcs.component_name, h.host_name;
            host_name             |  service_name  |     component_name      |   version
----------------------------------+----------------+-------------------------+-------------
 scregione1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm0.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm2.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionw0.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionw1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm0.field.hortonworks.com | ATLAS          | ATLAS_SERVER            | 2.6.1.0-129
 scregionm1.field.hortonworks.com | DRUID          | DRUID_BROKER            | 2.6.2.0-205
 scregionm1.field.hortonworks.com | DRUID          | DRUID_COORDINATOR       | 2.6.2.0-205
 scregionw0.field.hortonworks.com | DRUID          | DRUID_HISTORICAL        | 2.6.2.0-205
 scregionw1.field.hortonworks.com | DRUID          | DRUID_HISTORICAL        | 2.6.2.0-205
 scregionw0.field.hortonworks.com | DRUID          | DRUID_MIDDLEMANAGER     | 2.6.2.0-205
 scregionw1.field.hortonworks.com | DRUID          | DRUID_MIDDLEMANAGER     | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_OVERLORD          | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_ROUTER            | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_SUPERSET          | 2.6.2.0-205
 scregione1.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
 scregionm0.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
 scregionm1.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
. . .

After verifying that you have, indeed, upgraded the components, a simple update command will set the proper version for the erroneous components and allow you to finalize the upgrade:

ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_CLIENT';
UPDATE 6
ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_SERVER';
UPDATE 1

After cycling the Ambari Server, you should be able to finalize:

[root@hostname ~]# ambari-server set-current --cluster=<cluster> --version-display-name=HDP-2.6.2.0
Using python  /usr/bin/python
Setting current version...
Enter Ambari Admin login: <username>
Enter Ambari Admin password:


Current version successfully updated to HDP-2.6.2.0
Ambari Server 'set-current' completed successfully.

View solution in original post

4,912 Views
7 REPLIES 7

avatar
author-rank akhilsnaik author-rank
Guru

Created ‎10-09-2017 05:32 AM

Hi @dsun,

you can see the Status of upgrade in Database,

psql ambari -U ambari

default password is bigdata

in DB execute the following command to see package versions ( to know status of upgrade)

select * from hostcomponentstate;

 id  | cluster_id |     component_name      |   version   | current_stack_id | current_state | host_id |  service_name  | upgrade_state | security_state
-----+------------+-------------------------+-------------+------------------+---------------+---------+----------------+---------------+----------------
  24 |          2 | HDFS_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       1 | HDFS           | NONE          | UNSECURED
  21 |          2 | DATANODE                | 2.6.2.0-205 |                7 | STARTED       |       2 | HDFS           | NONE          | UNSECURED
  26 |          2 | HDFS_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       4 | HDFS           | NONE          | UNSECURED
   1 |          2 | NAMENODE                | 2.6.2.0-205 |                7 | STARTED       |       4 | HDFS           | NONE          | UNSECURED
  30 |          2 | MAPREDUCE2_CLIENT       | 2.6.2.0-205 |                7 | INSTALLED     |       4 | MAPREDUCE2     | NONE          | UNSECURED
   2 |          2 | SECONDARY_NAMENODE      | 2.6.2.0-205 |                7 | STARTED       |       1 | HDFS           | NONE          | UNSECURED
 101 |          2 | OOZIE_SERVER            | 2.6.2.0-205 |                7 | STARTED       |       1 | OOZIE          | NONE          | UNSECURED
  25 |          2 | HDFS_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       2 | HDFS           | NONE          | UNSECURED
  34 |          2 | TEZ_CLIENT              | 2.6.2.0-205 |                7 | INSTALLED     |       4 | TEZ            | NONE          | UNSECURED
 103 |          2 | OOZIE_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       2 | OOZIE          | NONE          | UNSECURED
 102 |          2 | OOZIE_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       1 | OOZIE          | NONE          | UNSECURED
  52 |          2 | METRICS_MONITOR         | UNKNOWN     |                7 | STARTED       |       1 | AMBARI_METRICS | NONE          | UNSECURED
   5 |          2 | HISTORYSERVER           | 2.6.2.0-205 |                7 | STARTED       |       1 | MAPREDUCE2     | NONE          | UNSECURED
   4 |          2 | APP_TIMELINE_SERVER     | 2.6.2.0-205 |                7 | STARTED       |       1 | YARN           | NONE          | UNSECURED
  58 |          2 | HST_AGENT               | UNKNOWN     |                7 | INSTALLED     |       1 | SMARTSENSE     | NONE          | UNSECURED
  39 |          2 | HBASE_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       2 | HBASE          | NONE          | UNSECURED
  17 |          2 | HST_SERVER              | UNKNOWN     |                7 | STARTED       |       4 | SMARTSENSE     | NONE          | UNSECURED
  59 |          2 | MYSQL_SERVER            | UNKNOWN     |                7 | STARTED       |       1 | HIVE           | NONE          | UNSECURED
  12 |          2 | ZOOKEEPER_SERVER        | 2.6.2.0-205 |                7 | STARTED       |       4 | ZOOKEEPER      | NONE          | UNSECURED
  51 |          2 | METRICS_MONITOR         | UNKNOWN     |                7 | STARTED       |       4 | AMBARI_METRICS | NONE          | UNSECURED
   3 |          2 | RESOURCEMANAGER         | 2.6.2.0-205 |                7 | STARTED       |       1 | YARN           | NONE          | UNSECURED
  45 |          2 | INFRA_SOLR_CLIENT       | UNKNOWN     |                7 | INSTALLED     |       4 | AMBARI_INFRA   | NONE          | UNSECURED
  18 |          2 | ACTIVITY_ANALYZER       | UNKNOWN     |                7 | INSTALLED     |       4 | SMARTSENSE     | NONE          | UNSECURED
  38 |          2 | HIVE_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       1 | HIVE           | NONE          | UNSECURED
  54 |          2 | METRICS_MONITOR         | UNKNOWN     |                7 | STARTED       |       2 | AMBARI_METRICS | NONE          | UNSECURED
  55 |          2 | HST_AGENT               | UNKNOWN     |                7 | INSTALLED     |       4 | SMARTSENSE     | NONE          | UNSECURED
  13 |          2 | INFRA_SOLR              | UNKNOWN     |                7 | STARTED       |       4 | AMBARI_INFRA   | NONE          | UNSECURED
  31 |          2 | MAPREDUCE2_CLIENT       | 2.6.2.0-205 |                7 | INSTALLED     |       1 | MAPREDUCE2     | NONE          | UNSECURED
  11 |          2 | ZOOKEEPER_SERVER        | 2.6.2.0-205 |                7 | STARTED       |       3 | ZOOKEEPER      | NONE          | UNSECURED
  14 |          2 | METRICS_GRAFANA         | UNKNOWN     |                7 | STARTED       |       4 | AMBARI_METRICS | NONE          | UNSECURED
  53 |          2 | METRICS_MONITOR         | UNKNOWN     |                7 | STARTED       |       3 | AMBARI_METRICS | NONE          | UNSECURED
  44 |          2 | INFRA_SOLR_CLIENT       | UNKNOWN     |                7 | INSTALLED     |       2 | AMBARI_INFRA   | NONE          | UNSECURED
  36 |          2 | HCAT                    | 2.6.2.0-205 |                7 | INSTALLED     |       2 | HIVE           | NONE          | UNSECURED
  37 |          2 | HIVE_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       2 | HIVE           | NONE          | UNSECURED
  32 |          2 | MAPREDUCE2_CLIENT       | 2.6.2.0-205 |                7 | INSTALLED     |       2 | MAPREDUCE2     | NONE          | UNSECURED
  20 |          2 | SPARK2_JOBHISTORYSERVER | 2.6.2.0-205 |                7 | STARTED       |       4 | SPARK2         | NONE          | UNSECURED
  15 |          2 | METRICS_COLLECTOR       | UNKNOWN     |                7 | STARTED       |       3 | AMBARI_METRICS | NONE          | UNSECURED
  23 |          2 | HBASE_REGIONSERVER      | 2.6.2.0-205 |                7 | STARTED       |       2 | HBASE          | NONE          | UNSECURED
  50 |          2 | SLIDER                  | 2.6.2.0-205 |                7 | INSTALLED     |       2 | SLIDER         | NONE          | UNSECURED
   6 |          2 | HIVE_METASTORE          | 2.6.2.0-205 |                7 | STARTED       |       1 | HIVE           | NONE          | UNSECURED
  41 |          2 | PIG                     | 2.6.2.0-205 |                7 | INSTALLED     |       2 | PIG            | NONE          | UNSECURED
  27 |          2 | YARN_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       4 | YARN           | NONE          | UNSECURED
  40 |          2 | PIG                     | 2.6.2.0-205 |                7 | INSTALLED     |       1 | PIG            | NONE          | UNSECURED
  49 |          2 | SLIDER                  | 2.6.2.0-205 |                7 | INSTALLED     |       1 | SLIDER         | NONE          | UNSECURED
  46 |          2 | SPARK_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       1 | SPARK          | NONE          | UNSECURED
  10 |          2 | ZOOKEEPER_SERVER        | 2.6.2.0-205 |                7 | STARTED       |       1 | ZOOKEEPER      | NONE          | UNSECURED
   9 |          2 | HBASE_MASTER            | 2.6.2.0-205 |                7 | STARTED       |       4 | HBASE          | NONE          | UNSECURED
  48 |          2 | SPARK2_CLIENT           | 2.6.2.0-205 |                7 | INSTALLED     |       2 | SPARK2         | NONE          | UNSECURED
  35 |          2 | TEZ_CLIENT              | 2.6.2.0-205 |                7 | INSTALLED     |       1 | TEZ            | NONE          | UNSECURED
  22 |          2 | NODEMANAGER             | 2.6.2.0-205 |                7 | STARTED       |       2 | YARN           | NONE          | UNSECURED
  28 |          2 | YARN_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       1 | YARN           | NONE          | UNSECURED
  33 |          2 | TEZ_CLIENT              | 2.6.2.0-205 |                7 | INSTALLED     |       2 | TEZ            | NONE          | UNSECURED
  42 |          2 | ZOOKEEPER_CLIENT        | 2.6.2.0-205 |                7 | INSTALLED     |       1 | ZOOKEEPER      | NONE          | UNSECURED
  43 |          2 | ZOOKEEPER_CLIENT        | 2.6.2.0-205 |                7 | INSTALLED     |       2 | ZOOKEEPER      | NONE          | UNSECURED
  47 |          2 | SPARK_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       2 | SPARK          | NONE          | UNSECURED
  57 |          2 | HST_AGENT               | UNKNOWN     |                7 | INSTALLED     |       3 | SMARTSENSE     | NONE          | UNSECURED
  29 |          2 | YARN_CLIENT             | 2.6.2.0-205 |                7 | INSTALLED     |       2 | YARN           | NONE          | UNSECURED
  56 |          2 | HST_AGENT               | UNKNOWN     |                7 | INSTALLED     |       2 | SMARTSENSE     | NONE          | UNSECURED
  16 |          2 | ACTIVITY_EXPLORER       | UNKNOWN     |                7 | STARTED       |       4 | SMARTSENSE     | NONE          | UNSECURED
 104 |          2 | OOZIE_CLIENT            | 2.6.2.0-205 |                7 | INSTALLED     |       4 | OOZIE          | NONE          | UNSECURED
  19 |          2 | SPARK_JOBHISTORYSERVER  | 2.6.2.0-205 |                7 | STARTED       |       4 | SPARK          | NONE          | UNSECURED
   7 |          2 | WEBHCAT_SERVER          | 2.6.2.0-205 |                7 | STARTED       |       1 | HIVE           | NONE          | UNSECURED
   8 |          2 | HIVE_SERVER             | 2.6.2.0-205 |                7 | STARTED       |       1 | HIVE           | NONE          | UNSECURED

select * from hostcomponentdesiredstate;
id  | cluster_id |     component_name      | desired_stack_id | desired_state | host_id |  service_name  | admin_state | maintenance_state | security_state | restart_required
-----+------------+-------------------------+------------------+---------------+---------+----------------+-------------+-------------------+----------------+------------------
   6 |          2 | HIVE_METASTORE          |                7 | STARTED       |       1 | HIVE           |             | OFF               | UNSECURED      |                0
   7 |          2 | WEBHCAT_SERVER          |                7 | STARTED       |       1 | HIVE           |             | OFF               | UNSECURED      |                0
  19 |          2 | SPARK_JOBHISTORYSERVER  |                7 | STARTED       |       4 | SPARK          |             | OFF               | UNSECURED      |                0
  20 |          2 | SPARK2_JOBHISTORYSERVER |                7 | STARTED       |       4 | SPARK2         |             | OFF               | UNSECURED      |                0
  23 |          2 | HBASE_REGIONSERVER      |                7 | STARTED       |       2 | HBASE          | INSERVICE   | OFF               | UNSECURED      |                0
  51 |          2 | METRICS_MONITOR         |                7 | STARTED       |       4 | AMBARI_METRICS | INSERVICE   | OFF               | UNSECURED      |                0
  52 |          2 | METRICS_MONITOR         |                7 | STARTED       |       1 | AMBARI_METRICS | INSERVICE   | OFF               | UNSECURED      |                0
  53 |          2 | METRICS_MONITOR         |                7 | STARTED       |       3 | AMBARI_METRICS | INSERVICE   | OFF               | UNSECURED      |                0
  54 |          2 | METRICS_MONITOR         |                7 | STARTED       |       2 | AMBARI_METRICS | INSERVICE   | OFF               | UNSECURED      |                0
  55 |          2 | HST_AGENT               |                7 | STARTED       |       4 | SMARTSENSE     | INSERVICE   | OFF               | UNSECURED      |                0
  56 |          2 | HST_AGENT               |                7 | STARTED       |       2 | SMARTSENSE     | INSERVICE   | OFF               | UNSECURED      |                0
  57 |          2 | HST_AGENT               |                7 | STARTED       |       3 | SMARTSENSE     | INSERVICE   | OFF               | UNSECURED      |                0
  58 |          2 | HST_AGENT               |                7 | STARTED       |       1 | SMARTSENSE     | INSERVICE   | OFF               | UNSECURED      |                0
  59 |          2 | MYSQL_SERVER            |                7 | STARTED       |       1 | HIVE           |             | OFF               | UNSECURED      |                0
 102 |          2 | OOZIE_CLIENT            |                7 | INSTALLED     |       1 | OOZIE          |             | OFF               | UNSECURED      |                0
 104 |          2 | OOZIE_CLIENT            |                7 | INSTALLED     |       4 | OOZIE          |             | OFF               | UNSECURED      |                0
   1 |          2 | NAMENODE                |                7 | STARTED       |       4 | HDFS           |             | OFF               | UNSECURED      |                0
   2 |          2 | SECONDARY_NAMENODE      |                7 | STARTED       |       1 | HDFS           |             | OFF               | UNSECURED      |                0
   3 |          2 | RESOURCEMANAGER         |                7 | STARTED       |       1 | YARN           |             | OFF               | UNSECURED      |                0
   4 |          2 | APP_TIMELINE_SERVER     |                7 | STARTED       |       1 | YARN           |             | OFF               | UNSECURED      |                0
   5 |          2 | HISTORYSERVER           |                7 | STARTED       |       1 | MAPREDUCE2     |             | OFF               | UNSECURED      |                0
  21 |          2 | DATANODE                |                7 | STARTED       |       2 | HDFS           | INSERVICE   | OFF               | UNSECURED      |                0
  22 |          2 | NODEMANAGER             |                7 | STARTED       |       2 | YARN           | INSERVICE   | OFF               | UNSECURED      |                0
  24 |          2 | HDFS_CLIENT             |                7 | INSTALLED     |       1 | HDFS           |             | OFF               | UNSECURED      |                0
  25 |          2 | HDFS_CLIENT             |                7 | INSTALLED     |       2 | HDFS           |             | OFF               | UNSECURED      |                0
  26 |          2 | HDFS_CLIENT             |                7 | INSTALLED     |       4 | HDFS           |             | OFF               | UNSECURED      |                0
  27 |          2 | YARN_CLIENT             |                7 | INSTALLED     |       4 | YARN           |             | OFF               | UNSECURED      |                0
  28 |          2 | YARN_CLIENT             |                7 | INSTALLED     |       1 | YARN           |             | OFF               | UNSECURED      |                0
  29 |          2 | YARN_CLIENT             |                7 | INSTALLED     |       2 | YARN           |             | OFF               | UNSECURED      |                0
  30 |          2 | MAPREDUCE2_CLIENT       |                7 | INSTALLED     |       4 | MAPREDUCE2     |             | OFF               | UNSECURED      |                0
  31 |          2 | MAPREDUCE2_CLIENT       |                7 | INSTALLED     |       1 | MAPREDUCE2     |             | OFF               | UNSECURED      |                0
  32 |          2 | MAPREDUCE2_CLIENT       |                7 | INSTALLED     |       2 | MAPREDUCE2     |             | OFF               | UNSECURED      |                0
  33 |          2 | TEZ_CLIENT              |                7 | INSTALLED     |       2 | TEZ            |             | OFF               | UNSECURED      |                0
  34 |          2 | TEZ_CLIENT              |                7 | INSTALLED     |       4 | TEZ            |             | OFF               | UNSECURED      |                0
  35 |          2 | TEZ_CLIENT              |                7 | INSTALLED     |       1 | TEZ            |             | OFF               | UNSECURED      |                0
  36 |          2 | HCAT                    |                7 | INSTALLED     |       2 | HIVE           |             | OFF               | UNSECURED      |                0
  37 |          2 | HIVE_CLIENT             |                7 | INSTALLED     |       2 | HIVE           |             | OFF               | UNSECURED      |                0
  38 |          2 | HIVE_CLIENT             |                7 | INSTALLED     |       1 | HIVE           |             | OFF               | UNSECURED      |                0
  39 |          2 | HBASE_CLIENT            |                7 | INSTALLED     |       2 | HBASE          |             | OFF               | UNSECURED      |                0
  40 |          2 | PIG                     |                7 | INSTALLED     |       1 | PIG       
     |             | OFF               | UNSECURED      |                0
  41 |          2 | PIG                     |                7 | INSTALLED     |       2 | PIG            |             | OFF               | UNSECURED      |                0
  42 |          2 | ZOOKEEPER_CLIENT        |                7 | INSTALLED     |       1 | ZOOKEEPER      |             | OFF               | UNSECURED      |                0
  43 |          2 | ZOOKEEPER_CLIENT        |                7 | INSTALLED     |       2 | ZOOKEEPER      |             | OFF               | UNSECURED      |                0
  44 |          2 | INFRA_SOLR_CLIENT       |                7 | INSTALLED     |       2 | AMBARI_INFRA   |             | OFF               | UNSECURED      |                0
  45 |          2 | INFRA_SOLR_CLIENT       |                7 | INSTALLED     |       4 | AMBARI_INFRA   |             | OFF               | UNSECURED      |                0
  46 |          2 | SPARK_CLIENT            |                7 | INSTALLED     |       1 | SPARK          |             | OFF               | UNSECURED      |                0
  47 |          2 | SPARK_CLIENT            |                7 | INSTALLED     |       2 | SPARK          |             | OFF               | UNSECURED      |                0
  48 |          2 | SPARK2_CLIENT           |                7 | INSTALLED     |       2 | SPARK2         |             | OFF               | UNSECURED      |                0
  49 |          2 | SLIDER                  |                7 | INSTALLED     |       1 | SLIDER         |             | OFF               | UNSECURED      |                0
  50 |          2 | SLIDER                  |                7 | INSTALLED     |       2 | SLIDER         |             | OFF               | UNSECURED      |                0
   8 |          2 | HIVE_SERVER             |                7 | STARTED       |       1 | HIVE           |             | OFF               | UNSECURED      |                0
   9 |          2 | HBASE_MASTER            |                7 | STARTED       |       4 | HBASE          |             | OFF               | UNSECURED      |                0
  10 |          2 | ZOOKEEPER_SERVER        |                7 | STARTED       |       1 | ZOOKEEPER      |             | OFF               | UNSECURED      |                0
  11 |          2 | ZOOKEEPER_SERVER        |                7 | STARTED       |       3 | ZOOKEEPER      |             | OFF               | UNSECURED      |                0
  12 |          2 | ZOOKEEPER_SERVER        |                7 | STARTED       |       4 | ZOOKEEPER      |             | OFF               | UNSECURED      |                0
  13 |          2 | INFRA_SOLR              |                7 | STARTED       |       4 | AMBARI_INFRA   |             | OFF               | UNSECURED      |                0
  14 |          2 | METRICS_GRAFANA         |                7 | STARTED       |       4 | AMBARI_METRICS |             | OFF               | UNSECURED      |                0
  15 |          2 | METRICS_COLLECTOR       |                7 | STARTED       |       3 | AMBARI_METRICS |             | OFF               | UNSECURED      |                0
  16 |          2 | ACTIVITY_EXPLORER       |                7 | STARTED       |       4 | SMARTSENSE     |             | OFF               | UNSECURED      |                0
  17 |          2 | HST_SERVER              |                7 | STARTED       |       4 | SMARTSENSE     |             | OFF               | UNSECURED      |                0
  18 |          2 | ACTIVITY_ANALYZER       |                7 | STARTED       |       4 | SMARTSENSE     |             | OFF               | UNSECURED      |                0
 103 |          2 | OOZIE_CLIENT            |                7 | INSTALLED     |       2 | OOZIE          |             | OFF               | UNSECURED      |                0
 101 |          2 | OOZIE_SERVER            |                7 | STARTED       |       1 | OOZIE          |             | OFF               | UNSECURED      |                0


select * from host_version;

 id | repo_version_id | host_id |  state
----+-----------------+---------+---------
  2 |               1 |       3 | CURRENT
  3 |               1 |       4 | CURRENT
  4 |               1 |       1 | CURRENT
  1 |               1 |       2 | CURRENT

select * from cluster_version;

 id | repo_version_id | cluster_id |  state  |  start_time   |   end_time    | user_name
----+-----------------+------------+---------+---------------+---------------+-----------
  1 |               1 |          2 | CURRENT | 1505793403390 | 1505794154404 | admin

you can update the host_versions to UPGRADING incase some upgrades has not happened.

you can troubleshoot more based on output of this database commands.

4,912 Views

avatar
author-rank dsun author-rank
Expert Contributor

Created ‎10-10-2017 02:28 AM

@Akhil S Naik updating the state column from either 'CURRENT' or 'INSTALLED' to 'UPGRADING' is causing Ambari fails to start for me:

ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.

4,912 Views
0 Kudos

avatar
author-rank jsensharma author-rank
Master Mentor

Created ‎10-09-2017 05:59 AM

@dsun

Please try the following:


1). For the Missing Upgrade button issue, Please try the following URL to open ambari in Experimental mode and check the "http://$AMBARI_HOST:8080/#/experimental" and save it in the UI to see if it helps:

opsDuringRollingUpgrade   = true (Checked)



2). Regarding the DRUID upgrade issue. Please login to the host "scregionm2.field.hortonworks.com" via SSH and then check the following:

a). If the Druid package is upgraded?

b). Check if all the verison are reflecting "2.6.2.0-205" version, in the output of the hdp-select output. If all installed components are correctly showing "2.6.2.0-205" (except DRUID) then refere to Step c.

# hdp-select

c). If you have installed DRUID from some third party sources then run the following command to set it's version to 2.6.2.0-205
Syntax:

# hdp-select set <PACKAGE_NAME> 2.6.2.0-205


Here PACKAGE_NAME will be druid name.

(OR)

# hdp-select set all 2.6.2.0-205

.

4,912 Views
0 Kudos

avatar
author-rank dsun author-rank
Expert Contributor

Created on ‎10-10-2017 02:01 AM - edited ‎08-17-2019 08:14 PM

Checked "opsDuringRollingUpgrade =true", and still not seeing the resume upgrade button. Just to make myself clear, I can see the upgrade button, as shown. However, when I click it, and select express upgrade, it won't pass the check, and saying there is already an upgrade in process, but wouldn't let me resume the previous upgrade.

40764-upgrade.png

40765-missingresumeupgrade.png

4,912 Views
0 Kudos

avatar
author-rank jsensharma author-rank
Master Mentor

Created ‎10-10-2017 03:54 AM

@dsun

Please do not change any table state to "UPGRADING" that will cause issues.

As you are getting the following message:

Reason: There is an existing upgrade from 2.6.1.0-129 to 2.6.2.0-205 which has not completed. This upgrade must be completed before a new upgrade or downgrade can begin.

It will be best if you can share the output of the following tables queries which seems to be not alright:

ambari=> SELECT * FROM host_version;
ambari=> SELECT * FROM cluster_version;
ambari=> SELECT * FROM clusterstate;
ambari=> SELECT repo_version_id, stack_id, version FROm repo_version;

.

Can you please share the output of the above SQL statements?

4,912 Views
0 Kudos

avatar
author-rank dsun author-rank
Expert Contributor

Created on ‎10-10-2017 02:39 PM - edited ‎08-17-2019 08:14 PM

Thanks @Jay SenSharma. Please see the output below:

40776-host-version.png

4,912 Views
0 Kudos

avatar
author-rank emaxwell author-rank
Guru

Created ‎10-19-2017 09:11 PM

@dsun

During the upgrade process, a component is supposed to be restarted after the hdp-select command has been run so it will pick up the new binaries. However, the component needs to shut down and start up after the hdp-select command has been run. That way it will report to Ambari that it's version has changed and what it's current state is. In the event that you get stuck (as you did) during the upgrade you can unwind the versioning with a process like this:

  1. Make all pieces of the component are running
  2. Run `hdp-select set` command on all nodes in the cluster to set the new version. Make sure you get all of the pieces for the component (e.g. hadoop-hdfs-namenode, hadoop-ndfs-journalnode, etc.)
  3. Restart all processes for the component
  4. Verify that the O/S processes are running with the proper version of jar files
  5. Lather, rinse, and repeat for all components in the cluster

Once you have successfully gotten everything restarted with the proper bits, you should be able to manually finalize the upgrade with the following command to the Ambari Server:

ambari-server set-current --cluster=<custername> --version-display-name=HDP-2.6.2.0

If you get an error that components are not upgraded, you can check the components and hosts again. If everything seems ok, then you may need to tweak a table in the database. I ran into this when Atlas did not properly report the upgraded version to Ambari. NOTE: THIS SHOULD BE DONE WITH THE GUIDANCE OF HORTONWORKS SUPPORT ONLY

ambari=> SELECT h.host_name, hcs.service_name, hcs.component_name, hcs.version FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id ORDER BY hcs.version, hcs.service_name, hcs.component_name, h.host_name;
            host_name             |  service_name  |     component_name      |   version
----------------------------------+----------------+-------------------------+-------------
 scregione1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm0.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm2.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionw0.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionw1.field.hortonworks.com | ATLAS          | ATLAS_CLIENT            | 2.6.1.0-129
 scregionm0.field.hortonworks.com | ATLAS          | ATLAS_SERVER            | 2.6.1.0-129
 scregionm1.field.hortonworks.com | DRUID          | DRUID_BROKER            | 2.6.2.0-205
 scregionm1.field.hortonworks.com | DRUID          | DRUID_COORDINATOR       | 2.6.2.0-205
 scregionw0.field.hortonworks.com | DRUID          | DRUID_HISTORICAL        | 2.6.2.0-205
 scregionw1.field.hortonworks.com | DRUID          | DRUID_HISTORICAL        | 2.6.2.0-205
 scregionw0.field.hortonworks.com | DRUID          | DRUID_MIDDLEMANAGER     | 2.6.2.0-205
 scregionw1.field.hortonworks.com | DRUID          | DRUID_MIDDLEMANAGER     | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_OVERLORD          | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_ROUTER            | 2.6.2.0-205
 scregionm2.field.hortonworks.com | DRUID          | DRUID_SUPERSET          | 2.6.2.0-205
 scregione1.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
 scregionm0.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
 scregionm1.field.hortonworks.com | HBASE          | HBASE_CLIENT            | 2.6.2.0-205
. . .

After verifying that you have, indeed, upgraded the components, a simple update command will set the proper version for the erroneous components and allow you to finalize the upgrade:

ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_CLIENT';
UPDATE 6
ambari=> update hostcomponentstate set version='2.6.2.0-205' where component_name = 'ATLAS_SERVER';
UPDATE 1

After cycling the Ambari Server, you should be able to finalize:

[root@hostname ~]# ambari-server set-current --cluster=<cluster> --version-display-name=HDP-2.6.2.0
Using python  /usr/bin/python
Setting current version...
Enter Ambari Admin login: <username>
Enter Ambari Admin password:


Current version successfully updated to HDP-2.6.2.0
Ambari Server 'set-current' completed successfully.
4,913 Views
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements