Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

mkukreja · ‎04-26-2019

Hello,

I am upgrading ambari 2.6.1.5-3 => 2.7.3.0, rpm install went fine and then I did

# ambari-server upgrade -v

# ambari-server upgrade -v

Using python /usr/bin/python

Upgrading ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrade Ambari Server

INFO: Updating Ambari Server properties in ambari.properties ...

WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings

INFO: Updating Ambari Server properties in ambari-env.sh ...

INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: No mpack replay logs found. Skipping replaying mpack commands

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Fixing database objects owner

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrading database schema

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1

INFO:

process_pid=10477

Traceback (most recent call last):

File "/usr/sbin/ambari-server.py", line 1060, in <module>

mainBody()

File "/usr/sbin/ambari-server.py", line 1030, in mainBody

main(options, args, parser)

File "/usr/sbin/ambari-server.py", line 980, in main

action_obj.execute()

File "/usr/sbin/ambari-server.py", line 77, in execute

self.fn(*self.args, **self.kwargs)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade

retcode = run_schema_upgrade(args)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade

upgrade_response = json.loads(stdout)

File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads

return _default_decoder.decode(s)

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode

raise ValueError("No JSON object could be decoded")

ValueError: No JSON object could be decoded

/var/log/ambari-server/ambari-server.log shows following errors:

# grep -i error ambari-server.log

2019-04-25 16:28:03,205 ERROR [main] StackContext:189 - Could not load repo results

2019-04-25 16:28:03,528 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint

2019-04-25 16:28:03,530 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint

2019-04-25 16:28:03,534 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint

2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.

2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed

Please help.

Thanks

Shelton · ‎04-26-2019

@Mahendra Kukreja

Backup your Ambari DB

mysqldump -u root -p ambari > ambari26042019_backup.sql

Check the indexes

SHOW INDEX FROM `hostcomponentdesiredstate`; SHOW INDEX FROM `hostcomponentstate`; SHOW INDEX FROM `hservicecomponentdesiredstate`;

ALTER TABLE `hostcomponentdesiredstate` DISABLE KEYS; ALTER TABLE `hostcomponentstate` DISABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DISABLE KEYS;

Then after the upgrade:

ALTER TABLE `hostcomponentdesiredstate` ENABLE KEYS; ALTER TABLE `hostcomponentstate` ENABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DENABLE KEYS;

If you use InnoDB tables instead of MyISAM tables, add these extra commands at the top of the file to avoid a great deal of disk access:

SET FOREIGN_KEY_CHECKS = 0; SET UNIQUE_CHECKS = 0; SET AUTOCOMMIT = 0;

And add these at the end:

SET UNIQUE_CHECKS = 1; SET FOREIGN_KEY_CHECKS = 1; COMMIT;

Restoration incase of failed upgrade case

mysqldump -u root -p ambari < ambari26042019_backup.sql

HTH

mkukreja · ‎04-27-2019

Thanks @Geoffrey Shelton Okot

My host is behind a firewall and internet access is only through a proxy so I configured my yum to use proxy for ambari repo if it wants to go out to internet, so now repo error is resolved but now I am getting following error:

# ambari-server upgrade -v

================================

Using python /usr/bin/python

Upgrading ambari-server