Created 04-26-2019 04:04 AM
Hello,
I am upgrading ambari 2.6.1.5-3 => 2.7.3.0, rpm install went fine and then I did
# ambari-server upgrade -v
# ambari-server upgrade -v Using python /usr/bin/python Upgrading ambari-server INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Upgrade Ambari Server INFO: Updating Ambari Server properties in ambari.properties ... WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings INFO: Updating Ambari Server properties in ambari-env.sh ... INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization. INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: No mpack replay logs found. Skipping replaying mpack commands INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Fixing database objects owner INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Upgrading database schema INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1 INFO: process_pid=10477 Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 1060, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 1030, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 980, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 77, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade retcode = run_schema_upgrade(args) File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade upgrade_response = json.loads(stdout) File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
/var/log/ambari-server/ambari-server.log
shows following errors:
# grep -i error ambari-server.log 2019-04-25 16:28:03,205 ERROR [main] StackContext:189 - Could not load repo results 2019-04-25 16:28:03,528 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint 2019-04-25 16:28:03,530 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint 2019-04-25 16:28:03,534 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint 2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed. 2019-04-25 16:28:12,259 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed
Please help.
Thanks
Created 04-26-2019 08:35 AM
Backup your Ambari DB
mysqldump -u root -p ambari > ambari26042019_backup.sql
Check the indexes
SHOW INDEX FROM `hostcomponentdesiredstate`; SHOW INDEX FROM `hostcomponentstate`; SHOW INDEX FROM `hservicecomponentdesiredstate`;
ALTER TABLE `hostcomponentdesiredstate` DISABLE KEYS; ALTER TABLE `hostcomponentstate` DISABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DISABLE KEYS;
Then after the upgrade:
ALTER TABLE `hostcomponentdesiredstate` ENABLE KEYS; ALTER TABLE `hostcomponentstate` ENABLE KEYS; ALTER TABLE `servicecomponentdesiredstate` DENABLE KEYS;
If you use InnoDB tables instead of MyISAM tables, add these extra commands at the top of the file to avoid a great deal of disk access:
SET FOREIGN_KEY_CHECKS = 0; SET UNIQUE_CHECKS = 0; SET AUTOCOMMIT = 0;
And add these at the end:
SET UNIQUE_CHECKS = 1; SET FOREIGN_KEY_CHECKS = 1; COMMIT;
Restoration incase of failed upgrade case
mysqldump -u root -p ambari < ambari26042019_backup.sql
HTH
Created 04-27-2019 03:51 PM
Thanks @Geoffrey Shelton Okot
My host is behind a firewall and internet access is only through a proxy so I configured my yum to use proxy for ambari repo if it wants to go out to internet, so now repo error is resolved but now I am getting following error:
# ambari-server upgrade -v
================================
Using python /usr/bin/python
Upgrading ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: No mpack replay logs found. Skipping replaying mpack commands
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Fixing database objects owner
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrading database schema
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1
INFO:
process_pid=27976
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 1060, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 1030, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 980, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 77, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade
retcode = run_schema_upgrade(args)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade
upgrade_response = json.loads(stdout)
File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
================================
# egrep -i 'warn|error' ambari-server.log (And some other relevant sections)
================================
2019-04-26 09:28:20,233 INFO [main] PasswordUtils:175 - Credential provider creation failed
org.apache.ambari.server.AmbariException: Master key initialization failed.
at org.apache.ambari.server.security.encryption.CredentialProvider.<init>(CredentialProvider.java:63)
at org.apache.ambari.server.utils.PasswordUtils.loadCredentialProvider(PasswordUtils.java:173)
at org.apache.ambari.server.utils.PasswordUtils.readPasswordFromStore(PasswordUtils.java:148)
at org.apache.ambari.server.configuration.Configuration.loadSSLParams(Configuration.java:2928)
at org.apache.ambari.server.configuration.Configuration.<init>(Configuration.java:2871)
at org.apache.ambari.server.configuration.Configuration.<init>(Configuration.java:2743)
at org.apache.ambari.server.controller.ControllerModule.<init>(ControllerModule.java:201)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper$UpgradeHelperModule.<init>(SchemaUpgradeHelper.java:173)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:408)
:
:
2019-04-26 09:28:22,088 INFO [main] ControllerModule:718 - Registered pre-upgrade check class org.apache.ambari.server.checks.HiveNotRollingWarning
2019-04-26 09:28:22,088 INFO [main] ControllerModule:718 - Registered pre-upgrade check class org.apache.ambari.server.checks.StormShutdownWarning
2019-04-26 09:28:29,319 ERROR [main] StackContext:189 - Could not load repo results
2019-04-26 09:28:29,618 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentdesiredstate_component_name not found for table hostcomponentdesiredstate
2019-04-26 09:28:29,618 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentdesiredstate_component_name from hostcomponentdesiredstate table does not exist and will not be dropped
2019-04-26 09:28:29,621 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint
2019-04-26 09:28:29,622 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentstate_component_name not found for table hostcomponentstate
2019-04-26 09:28:29,622 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentstate_component_name from hostcomponentstate table does not exist and will not be dropped
2019-04-26 09:28:29,624 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint
2019-04-26 09:28:29,625 WARN [main] DBAccessorImpl:382 - FK fk_servicecomponentdesiredstate_service_name not found for table servicecomponentdesiredstate
2019-04-26 09:28:29,625 WARN [main] DBAccessorImpl:997 - Foreign key fk_servicecomponentdesiredstate_service_name from servicecomponentdesiredstate table does not exist and will not be dropped
2019-04-26 09:28:29,627 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint
2019-04-26 09:28:29,639 WARN [main] DBAccessorImpl:581 - Primary constraint PK_ambari_configuration not altered to table ambari_configuration as column [category_name, property_name] not present or constraint already exists
2019-04-26 09:28:29,679 WARN [main] DBAccessorImpl:958 - members_tmp table doesn't exists, skipping
2019-04-26 09:28:29,689 WARN [main] DBAccessorImpl:958 - adminprivilege_tmp table doesn't exists, skipping
2019-04-26 09:28:29,828 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kerberos_keytab not altered to table kerberos_keytab as column [keytab_path] not present or constraint already exists
2019-04-26 09:28:29,829 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kkp not altered to table kerberos_keytab_principal as column [kkp_id] not present or constraint already exists
2019-04-26 09:28:29,830 WARN [main] DBAccessorImpl:581 - Primary constraint PK_kkp_mapping_service not altered to table kkp_mapping_service as column [kkp_id, service_name, component_name] not present or constraint already exists
2019-04-26 09:28:29,907 WARN [main] AbstractUpgradeCatalog:185 - Sequence kkp_id_seq already exists, skipping
2019-04-26 09:28:29,907 WARN [main] DBAccessorImpl:958 - kerberos_principal_host table doesn't exists, skipping
2019-04-26 09:28:29,908 WARN [main] DBAccessorImpl:581 - Primary constraint PK_repo_os_id not altered to table repo_os as column [id] not present or constraint already exists
2019-04-26 09:28:29,923 WARN [main] DBAccessorImpl:581 - Primary constraint PK_repo_definition_id not altered to table repo_definition as column [id] not present or constraint already exists
2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.
2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed
:
:
2019-04-26 09:28:38,193 INFO [main] UpgradeCatalog270:1724 - Upgrading 'authentication.ldap.primaryUrl' to 'ambari.ldap.connectivity.server.host'
2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
at com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.lambda$moveAmbariPropertiesToAmbariConfiguration$9(UpgradeCatalog270.java:1676)
at java.util.HashMap.forEach(HashMap.java:1288)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.moveAmbariPropertiesToAmbariConfiguration(UpgradeCatalog270.java:1666)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1061)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:238)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:458)
2019-04-26 09:28:38,193 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:458)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
at com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.lambda$moveAmbariPropertiesToAmbariConfiguration$9(UpgradeCatalog270.java:1676)
at java.util.HashMap.forEach(HashMap.java:1288)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.moveAmbariPropertiesToAmbariConfiguration(UpgradeCatalog270.java:1666)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1061)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:238)
... 1 more
================================
My default mysql engine is InnoDB. I tried the 3 commands you gave above:
SHOW INDEX FROM hostcomponentdesiredstate; SHOW INDEX FROM hostcomponentstate;
Above 2 commands are successful and I get the output, however the following command is giving error:
mysql> SHOW INDEX FROM hservicecomponentdesiredstate;
ERROR 1146 (42S02): Table 'ambaridb.hservicecomponentdesiredstate' doesn't exist
Created 04-27-2019 03:51 PM
And I am using:
mysql-connector-java-5.1.25-3
mysql-community-server-5.7.23-1