upgrade ambari-server 2.5.2 to 2.6.1

[#1]

Hello All,

I've been stuck with upgrading many versions of ambari-server to finally get to the version 2.6.1.0; but kept failing.

The last stack is from ambari-server of 2.5.2 to 2.6.1 with the following error

Traceback (most recent call last):
  File "/usr/sbin/ambari-server.py", line 952, in <module>
    mainBody()
  File "/usr/sbin/ambari-server.py", line 922, in mainBody
    main(options, args, parser)
  File "/usr/sbin/ambari-server.py", line 874, in main
    action_obj.execute()
  File "/usr/sbin/ambari-server.py", line 78, in execute
    self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 258, in upgrade
    retcode = run_schema_upgrade(args)
  File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 159, in run_schema_upgrade
    upgrade_response = json.loads(stdout)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

23 Jan 2018 13:30:58,621  INFO [main] LockFactory:53 - Lock profiling is disabled
23 Jan 2018 13:30:58,634  INFO [main] SchemaUpgradeHelper:420 - Upgrading schema to target version = 2.6.1.0
23 Jan 2018 13:30:58,673  INFO [main] SchemaUpgradeHelper:429 - Upgrading schema from source version = 2.5.2
23 Jan 2018 13:30:58,677  INFO [main] SchemaUpgradeHelper:163 - Upgrade path: [{ upgradeCatalog: sourceVersion = 2.5.2, targetVersion = 2.6.0 }, { upgradeCatalog: sourceVersion = 2.6.0, targetVersion = 2.6.1 }, { upgradeCatalog: sourceVersion = null, targetVersion = 2.6.1 }, { upgradeCatalog: sourceVersion = null, targetVersion = 2.6.1 }]
23 Jan 2018 13:30:58,677  INFO [main] SchemaUpgradeHelper:201 - Executing DDL upgrade...
23 Jan 2018 13:30:58,678  INFO [main] DBAccessorImpl:874 - Executing query: ALTER SCHEMA ambari OWNER TO "ambari";
23 Jan 2018 13:30:58,679  INFO [main] DBAccessorImpl:874 - Executing query: ALTER ROLE "ambari" SET search_path to 'ambari';
23 Jan 2018 13:30:58,710 ERROR [main] SchemaUpgradeHelper:208 - Upgrade failed.
org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories.
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:540)
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:210)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:931)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:206)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:444)
23 Jan 2018 13:30:58,713 ERROR [main] SchemaUpgradeHelper:464 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories.
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:209)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:444)
Caused by: org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories.
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:540)
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:210)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:931)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:206)
        ... 1 more	                                                                    

[#2 · Accepted solution]

It seems like your cluster was not in a consistent state. Ambari needs to know what repository/version your cluster is "CURRENT" on. We track this via the cluster_version table. There will be a bunch of entries in here, one for each repository you've ever installed.

It's OK for most of them to be OUT_OF_SYNC - that's expected. However, exactly 1 must be "CURRENT". I would find your correct repository version from the repo_version table and then update the corresponding row in the cluster_version table.

Something like this:

UPDATE cluster_version SET state = 'CURRENT' WHERE repo_version_id = <some-id>;

[#3]

Thank you @Jonathan Hurley for your response and explanation.

Pardon my very limited knowledge of ambari and postgresql. I logged on to ambari postgres and see the below relationship, but how do I find out what's my repo_version_id?

ambari | repo_version| table | postgres
ambari | cluster_version | table | postgres

Thank you for your help.

[#4]

Let me see if I can help you through this. Can you perform the following query for me:

SELECT repo_version_id, version, display_name FROM repo_version ORDER BY version;

This will get you a list that looks something like:

 repo_version_id |   version    |   display_name
-----------------+--------------+------------------
               1 | 2.5.0.0-1237 | HDP-2.5.0.0-1237
             101 | 2.5.4.0-121  | HDP-2.5.4.0-121
              51 | 2.6.0.0-334  | HDP-2.6.0.0-334

Chances are the most recent version is the one that you're on (or are at least supposed to be on). In my case, this is ID 51. So, you would do:

UPDATE cluster_version SET state = 'CURRENT' WHERE repo_version_id = 51;

The upgrade should work now after making this kind of change.

[#5]

Thank you so very much @Jonathan Hurley

With your help, I got my id and set my current state. I can now proceed to try again with my upgrade.

ambari=> SELECT repo_version_id, version, display_name FROM repo_version ORDER BY version;
repo_version_id |version | display_name
-----------------+------------+--------------
1 | 2.5.3.0-37 | HDP-2.5.3.0
(1 row)
ambari=> UPDATE cluster_version SET state = 'CURRENT' WHERE repo_version_id = 1;
UPDATE 1

Thank you!

[#6]

I was able to upgrade ambari to 2.6.1 successfully. Thank you again for your help @Jonathan Hurley