Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

Contributor

I am currently in the process of upgrading a 2 node cluster, and my cluster is currently stuck at 'Upgrade: In Process'. I have read a significant amount and tried many things, all of which have not helped. The first thing I check was the Ambari server logs, but nothing useful stood out.

I executed hdp-select set all 2.4.3.0-227 and it completes, but doesn't solve the problem.

I executed ambari-server set-current --cluster-name=MTNSA_HDP_DEV --version-display-name=HDP-2.4.3.0 based on this link. This gave the following output:

message" : "org.apache.ambari.server.controller.spi.SystemException: Finalization failed. More details: 
STDOUT: Begin finalizing the upgrade of cluster CLUSTER to version 2.4.3.0-227
STDERR: The following 52 host component(s) have not been upgraded to version 2.4.3.0-227. Please install and upgrade the Stack Version on those hosts and try again.
Host components:
SQOOP on host host01
SQOOP on host host02
SECONDARY_NAMENODE on host host01
HDFS_CLIENT on host host01
HDFS_CLIENT on host host02
DATANODE on host host02
NAMENODE on host host01
HISTORYSERVER on host host01
MAPREDUCE2_CLIENT on host host01
MAPREDUCE2_CLIENT on host host02
OOZIE_SERVER on host host01
OOZIE_CLIENT on host host01
OOZIE_CLIENT on host host02
TEZ_CLIENT on host host01
TEZ_CLIENT on host host02
SPARK_CLIENT on host host01
SPARK_CLIENT on host host02
SPARK_JOBHISTORYSERVER on host host01
HBASE_CLIENT on host host01
HBASE_CLIENT on host host02
HBASE_MASTER on host host01
HBASE_REGIONSERVER on host host02
PHOENIX_QUERY_SERVER on host host02
ZOOKEEPER_SERVER on host host01
ZOOKEEPER_CLIENT on host host01
ZOOKEEPER_CLIENT on host host02
NODEMANAGER on host host02
YARN_CLIENT on host host01
YARN_CLIENT on host host02
RESOURCEMANAGER on host host01
KNOX_GATEWAY on host host01
PIG on host host01
PIG on host host02
RANGER_ADMIN on host host01
RANGER_USERSYNC on host host01
SUPERVISOR on host host01
SUPERVISOR on host host02
NIMBUS on host host01
DRPC_SERVER on host host01
STORM_UI_SERVER on host host01
HIVE_SERVER on host host01
HIVE_METASTORE on host host01
HIVE_CLIENT on host host01
HIVE_CLIENT on host host02
HCAT on host host01
HCAT on host host02
WEBHCAT_SERVER on host host01
SLIDER on host host01
SLIDER on host host02
KAFKA_BROKER on host host01
FALCON_CLIENT on host host01
FALCON_CLIENT on host host02

(Apologies for long paste, I wanted to include anything useful). I navigated to http://host01:8080/api/v1/clusters/CLUSTER/upgrades and the result was

{
  "href" : "http://host01:8080/api/v1/clusters/CLUSTER/upgrades",
  "items" : [ ]
}

I also checked the Ambari DB with the following:

mysql> SELECT repo_version_id, version, display_name FROM repo_version;
+-----------------+-------------+-----------------+
| repo_version_id | version     | display_name    |
+-----------------+-------------+-----------------+
|               1 | 2.4.2.0-258 | HDP-2.4.2.0-258 |
|              51 | 2.4.3.0-227 | HDP-2.4.3.0     |
+-----------------+-------------+-----------------+
mysql> SELECT version, state, COUNT(*) FROM host_version hv JOIN repo_version rv ON hv.repo_version_id = rv.repo_version_id GROUP BY version, state ORDER BY version DESC, state;
+-------------+----------+----------+
| version     | state    | COUNT(*) |
+-------------+----------+----------+
| 2.4.3.0-227 | UPGRADED |        2 |
| 2.4.2.0-258 | CURRENT  |        2 |
+-------------+----------+----------+
mysql> SELECT service_name, component_name, version, host_name FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id WHERE service_name NOT IN ('AMBARI_METRICS', 'KERBEROS') and component_name NOT IN ('ZKFC') ORDER BY version, service_name, component_name, host_name;
+--------------+------------------------+-------------+-----------+
| service_name | component_name         | version     | host_name |
+--------------+------------------------+-------------+-----------+
| ATLAS        | ATLAS_SERVER           | 2.4.2.0-258 | host01    |
| FALCON       | FALCON_CLIENT          | 2.4.2.0-258 | host02    |
| HBASE        | HBASE_CLIENT           | 2.4.2.0-258 | host01    |
| HBASE        | HBASE_CLIENT           | 2.4.2.0-258 | host02    |
| HBASE        | HBASE_MASTER           | 2.4.2.0-258 | host01    |
| HBASE        | HBASE_REGIONSERVER     | 2.4.2.0-258 | host02    |
| HBASE        | PHOENIX_QUERY_SERVER   | 2.4.2.0-258 | host02    |
| HDFS         | DATANODE               | 2.4.2.0-258 | host02    |
| HDFS         | HDFS_CLIENT            | 2.4.2.0-258 | host01    |
| HDFS         | HDFS_CLIENT            | 2.4.2.0-258 | host02    |
| HDFS         | NAMENODE               | 2.4.2.0-258 | host01    |
| HDFS         | SECONDARY_NAMENODE     | 2.4.2.0-258 | host01    |
| HIVE         | HCAT                   | 2.4.2.0-258 | host01    |
| HIVE         | HCAT                   | 2.4.2.0-258 | host02    |
| HIVE         | HIVE_CLIENT            | 2.4.2.0-258 | host01    |
| HIVE         | HIVE_CLIENT            | 2.4.2.0-258 | host02    |
| HIVE         | HIVE_METASTORE         | 2.4.2.0-258 | host01    |
| HIVE         | HIVE_SERVER            | 2.4.2.0-258 | host01    |
| KAFKA        | KAFKA_BROKER           | 2.4.2.0-258 | host01    |
| KNOX         | KNOX_GATEWAY           | 2.4.2.0-258 | host01    |
| MAPREDUCE2   | HISTORYSERVER          | 2.4.2.0-258 | host01    |
| MAPREDUCE2   | MAPREDUCE2_CLIENT      | 2.4.2.0-258 | host01    |
| MAPREDUCE2   | MAPREDUCE2_CLIENT      | 2.4.2.0-258 | host02    |
| OOZIE        | OOZIE_CLIENT           | 2.4.2.0-258 | host01    |
| OOZIE        | OOZIE_CLIENT           | 2.4.2.0-258 | host02    |
| OOZIE        | OOZIE_SERVER           | 2.4.2.0-258 | host01    |
| PIG          | PIG                    | 2.4.2.0-258 | host01    |
| PIG          | PIG                    | 2.4.2.0-258 | host02    |
| RANGER       | RANGER_ADMIN           | 2.4.2.0-258 | host01    |
| RANGER       | RANGER_USERSYNC        | 2.4.2.0-258 | host01    |
| SLIDER       | SLIDER                 | 2.4.2.0-258 | host01    |
| SLIDER       | SLIDER                 | 2.4.2.0-258 | host02    |
| SPARK        | SPARK_CLIENT           | 2.4.2.0-258 | host01    |
| SPARK        | SPARK_CLIENT           | 2.4.2.0-258 | host02    |
| SPARK        | SPARK_JOBHISTORYSERVER | 2.4.2.0-258 | host01    |
| SQOOP        | SQOOP                  | 2.4.2.0-258 | host01    |
| SQOOP        | SQOOP                  | 2.4.2.0-258 | host02    |
| STORM        | DRPC_SERVER            | 2.4.2.0-258 | host01    |
| STORM        | NIMBUS                 | 2.4.2.0-258 | host01    |
| STORM        | STORM_UI_SERVER        | 2.4.2.0-258 | host01    |
| STORM        | SUPERVISOR             | 2.4.2.0-258 | host01    |
| STORM        | SUPERVISOR             | 2.4.2.0-258 | host02    |
| TEZ          | TEZ_CLIENT             | 2.4.2.0-258 | host01    |
| TEZ          | TEZ_CLIENT             | 2.4.2.0-258 | host02    |
| YARN         | APP_TIMELINE_SERVER    | 2.4.2.0-258 | host01    |
| YARN         | NODEMANAGER            | 2.4.2.0-258 | host02    |
| YARN         | RESOURCEMANAGER        | 2.4.2.0-258 | host01    |
| YARN         | YARN_CLIENT            | 2.4.2.0-258 | host01    |
| YARN         | YARN_CLIENT            | 2.4.2.0-258 | host02    |
| ZOOKEEPER    | ZOOKEEPER_CLIENT       | 2.4.2.0-258 | host01    |
| ZOOKEEPER    | ZOOKEEPER_CLIENT       | 2.4.2.0-258 | host02    |
| ZOOKEEPER    | ZOOKEEPER_SERVER       | 2.4.2.0-258 | host01    |
| FALCON       | FALCON_CLIENT          | 2.4.3.0-227 | host01    |
| FALCON       | FALCON_SERVER          | 2.4.3.0-227 | host01    |
| HIVE         | WEBHCAT_SERVER         | 2.4.3.0-227 | host01    |
+--------------+------------------------+-------------+-----------+

This indicates that all services apart from Falcon Server, one Falcon client and the WebHCat server are on the old version. What is confusing is that both hosts show 2.4.3 as UPGRADED, which should allow it to be finalized. I also checked /usr/hdp/current. All of these directories point to 2.4.3. How can I get my cluster to finalize on the upgrade?

Thanks in advance

4 REPLIES 4
Highlighted

Re: Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

Contributor

@Mark - Were there any failures during the upgrade? If yes, how did you fix them? Can you paste a screenshot of your Upgrade Wizard?

As for finalize, it would be permitted when Ambari knows that all components on all hosts have been successfully upgraded. You can hit 'Finalize Later' (equivalent to Pause Upgrade) and try the following:

  1. Run 'ambari-server restart' on Ambari server host
  2. Restart all services from Ambari UI
  3. Restart Ambari agents on both hosts

And then Resume upgrade and try to Finalize. This may/may not solve the issue depending on what the problem was during your upgrade

The other option would be to downgrade your cluster (since I see majority of the components do not show 2.4.3 version) and start upgrade afresh

-Vivek

Re: Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

@Mark Heydenrych

# What version do we see for falcon with the following command?

hdp-select  | grep falcon 

# If above still shows older version then try the following:

conf-select create-conf-dir --package falcon --stack-version 2.4.3.0-227 --conf-version 0 

conf-select set-conf-dir --package falcon --stack-version 2.4.3.0-227 --conf-version 0 

Same steps for the "webhcat" as well.

Re: Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

Contributor
hdp-select | grep falcon
falcon-client - 2.4.3.0-227
falcon-server - 2.4.3.0-227

Falcon is on the new version.

Re: Cannot Finalize Upgrade from HDP 2.4.2 to 2.4.3

Don't have an account?
Coming from Hortonworks? Activate your account here