Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

New Contributor

Hello,

I am upgrading ambari 2.6.1.5-3 => 2.7.3.0, rpm install went fine and then I did

# ambari-server upgrade -v

# ambari-server upgrade -v

Using python /usr/bin/python

Upgrading ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrade Ambari Server

INFO: Updating Ambari Server properties in ambari.properties ...

WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings

INFO: Updating Ambari Server properties in ambari-env.sh ...

INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: No mpack replay logs found. Skipping replaying mpack commands

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Fixing database objects owner

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrading database schema

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1

INFO:

process_pid=10477

Traceback (most recent call last):

File "/usr/sbin/ambari-server.py", line 1060, in <module>

mainBody()

File "/usr/sbin/ambari-server.py", line 1030, in mainBody

main(options, args, parser)

File "/usr/sbin/ambari-server.py", line 980, in main

action_obj.execute()

File "/usr/sbin/ambari-server.py", line 77, in execute

self.fn(*self.args, **self.kwargs)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade

retcode = run_schema_upgrade(args)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade

upgrade_response = json.loads(stdout)

File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads

return _default_decoder.decode(s)

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode

raise ValueError("No JSON object could be decoded")

ValueError: No JSON object could be decoded

/var/log/ambari-server/ambari-server.log shows following errors:

# grep -i error ambari-server.log

2019-04-25 16:28:03,205 ERROR [main] StackContext:189 - Could not load repo results

2019-04-25 16:28:03,528 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint

2019-04-25 16:28:03,530 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint

2019-04-25 16:28:03,534 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint

2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.

2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed


Please help.


Thanks


3 REPLIES 3

Re: Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

Mentor

@Mahendra Kukreja

Backup your Ambari DB

mysqldump -u root -p ambari > ambari26042019_backup.sql


Check the indexes

SHOW INDEX FROM `hostcomponentdesiredstate`; SHOW INDEX FROM `hostcomponentstate`; SHOW INDEX FROM `hservicecomponentdesiredstate`;


ALTER TABLE `hostcomponentdesiredstate` DISABLE KEYS; ALTER TABLE `hostcomponentstate` DISABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DISABLE KEYS;

Then after the upgrade:

ALTER TABLE `hostcomponentdesiredstate` ENABLE KEYS; ALTER TABLE `hostcomponentstate` ENABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DENABLE KEYS;

If you use InnoDB tables instead of MyISAM tables, add these extra commands at the top of the file to avoid a great deal of disk access:

SET FOREIGN_KEY_CHECKS = 0; SET UNIQUE_CHECKS = 0; SET AUTOCOMMIT = 0;

And add these at the end:

SET UNIQUE_CHECKS = 1; SET FOREIGN_KEY_CHECKS = 1; COMMIT;

Restoration incase of failed upgrade case

mysqldump -u root -p ambari < ambari26042019_backup.sql

HTH

Highlighted

Re: Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

New Contributor

Thanks @Geoffrey Shelton Okot

My host is behind a firewall and internet access is only through a proxy so I configured my yum to use proxy for ambari repo if it wants to go out to internet, so now repo error is resolved but now I am getting following error:

# ambari-server upgrade -v

================================

Using python /usr/bin/python

Upgrading ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrade Ambari Server

INFO: Updating Ambari Server properties in ambari.properties ...

WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings

INFO: Updating Ambari Server properties in ambari-env.sh ...

INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: No mpack replay logs found. Skipping replaying mpack commands

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Fixing database objects owner

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrading database schema

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1

INFO:

process_pid=27976

Traceback (most recent call last):

File "/usr/sbin/ambari-server.py", line 1060, in <module>

mainBody()

File "/usr/sbin/ambari-server.py", line 1030, in mainBody

main(options, args, parser)

File "/usr/sbin/ambari-server.py", line 980, in main

action_obj.execute()

File "/usr/sbin/ambari-server.py", line 77, in execute

self.fn(*self.args, **self.kwargs)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade

retcode = run_schema_upgrade(args)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade

upgrade_response = json.loads(stdout)

File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads

return _default_decoder.decode(s)

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode

raise ValueError("No JSON object could be decoded")

ValueError: No JSON object could be decoded

================================


# egrep -i 'warn|error' ambari-server.log (And some other relevant sections)

================================

2019-04-26 09:28:20,233 INFO [main] PasswordUtils:175 - Credential provider creation failed

org.apache.ambari.server.AmbariException: Master key initialization failed.

at org.apache.ambari.server.security.encryption.CredentialProvider.<init>(CredentialProvider.java:63)

at org.apache.ambari.server.utils.PasswordUtils.loadCredentialProvider(PasswordUtils.java:173)

at org.apache.ambari.server.utils.PasswordUtils.readPasswordFromStore(PasswordUtils.java:148)

at org.apache.ambari.server.configuration.Configuration.loadSSLParams(Configuration.java:2928)

at org.apache.ambari.server.configuration.Configuration.<init>(Configuration.java:2871)

at org.apache.ambari.server.configuration.Configuration.<init>(Configuration.java:2743)

at org.apache.ambari.server.controller.ControllerModule.<init>(ControllerModule.java:201)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper$UpgradeHelperModule.<init>(SchemaUpgradeHelper.java:173)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:408)


:

:

2019-04-26 09:28:22,088 INFO [main] ControllerModule:718 - Registered pre-upgrade check class org.apache.ambari.server.checks.HiveNotRollingWarning

2019-04-26 09:28:22,088 INFO [main] ControllerModule:718 - Registered pre-upgrade check class org.apache.ambari.server.checks.StormShutdownWarning

2019-04-26 09:28:29,319 ERROR [main] StackContext:189 - Could not load repo results

2019-04-26 09:28:29,618 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentdesiredstate_component_name not found for table hostcomponentdesiredstate

2019-04-26 09:28:29,618 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentdesiredstate_component_name from hostcomponentdesiredstate table does not exist and will not be dropped

2019-04-26 09:28:29,621 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint

2019-04-26 09:28:29,622 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentstate_component_name not found for table hostcomponentstate

2019-04-26 09:28:29,622 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentstate_component_name from hostcomponentstate table does not exist and will not be dropped

2019-04-26 09:28:29,624 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint

2019-04-26 09:28:29,625 WARN [main] DBAccessorImpl:382 - FK fk_servicecomponentdesiredstate_service_name not found for table servicecomponentdesiredstate

2019-04-26 09:28:29,625 WARN [main] DBAccessorImpl:997 - Foreign key fk_servicecomponentdesiredstate_service_name from servicecomponentdesiredstate table does not exist and will not be dropped

2019-04-26 09:28:29,627 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint

2019-04-26 09:28:29,639 WARN [main] DBAccessorImpl:581 - Primary constraint PK_ambari_configuration not altered to table ambari_configuration as column [category_name, property_name] not present or constraint already exists

2019-04-26 09:28:29,679 WARN [main] DBAccessorImpl:958 - members_tmp table doesn't exists, skipping

2019-04-26 09:28:29,689 WARN [main] DBAccessorImpl:958 - adminprivilege_tmp table doesn't exists, skipping

2019-04-26 09:28:29,828 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kerberos_keytab not altered to table kerberos_keytab as column [keytab_path] not present or constraint already exists

2019-04-26 09:28:29,829 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kkp not altered to table kerberos_keytab_principal as column [kkp_id] not present or constraint already exists

2019-04-26 09:28:29,830 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kkp_mapping_service not altered to table kkp_mapping_service as column [kkp_id, service_name, component_name] not present or constraint already exists

2019-04-26 09:28:29,907 WARN [main] AbstractUpgradeCatalog:185 - Sequence kkp_id_seq already exists, skipping

2019-04-26 09:28:29,907 WARN [main] DBAccessorImpl:958 - kerberos_principal_host table doesn't exists, skipping

2019-04-26 09:28:29,908 WARN [main] DBAccessorImpl:581 - Primary constraint PK_repo_os_id not altered to table repo_os as column [id] not present or constraint already exists

2019-04-26 09:28:29,923 WARN [main] DBAccessorImpl:581 - Primary constraint PK_repo_definition_id not altered to table repo_definition as column [id] not present or constraint already exists

2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.

2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed

:

:

2019-04-26 09:28:38,193 INFO [main] UpgradeCatalog270:1724 - Upgrading 'authentication.ldap.primaryUrl' to 'ambari.ldap.connectivity.server.host'

2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.

java.lang.IllegalStateException

at com.google.common.base.Preconditions.checkState(Preconditions.java:158)

at com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.lambda$moveAmbariPropertiesToAmbariConfiguration$9(UpgradeCatalog270.java:1676)

at java.util.HashMap.forEach(HashMap.java:1288)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.moveAmbariPropertiesToAmbariConfiguration(UpgradeCatalog270.java:1666)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1061)

at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:238)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:458)

2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed

org.apache.ambari.server.AmbariException

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:458)

Caused by: java.lang.IllegalStateException

at com.google.common.base.Preconditions.checkState(Preconditions.java:158)

at com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.lambda$moveAmbariPropertiesToAmbariConfiguration$9(UpgradeCatalog270.java:1676)

at java.util.HashMap.forEach(HashMap.java:1288)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.moveAmbariPropertiesToAmbariConfiguration(UpgradeCatalog270.java:1666)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1061)

at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:238)

... 1 more

================================


My default mysql engine is InnoDB. I tried the 3 commands you gave above:

SHOW INDEX FROM hostcomponentdesiredstate; SHOW INDEX FROM hostcomponentstate;

Above 2 commands are successful and I get the output, however the following command is giving error:

mysql> SHOW INDEX FROM hservicecomponentdesiredstate;

ERROR 1146 (42S02): Table 'ambaridb.hservicecomponentdesiredstate' doesn't exist


Re: Problem during ambari upgrade 2.6.1.5-3 => 2.7.3.0

New Contributor

And I am using:

mysql-connector-java-5.1.25-3

mysql-community-server-5.7.23-1