Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari 2.5.0 upgrade failed

Highlighted

Ambari 2.5.0 upgrade failed

Rising Star

I have a problem with upgrading Ambari from 2.4.0.1. to 2.5.0.

Actually my problems started from Ambari not start after downgrade

After dealing with Downgrade Can Create Multiple Mappings For Latest Configs I run Ambari as "ambari-server start --skip-database-check", fulfilled all prerequisites for Upgrade to 2.5 from instruction and finally on step

ambari-server upgrade

I got these errors.

Any ideas how to recover Ambari?

ambari-server.log

ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 4 INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (18, 403) was aborted.  Call getNextException to see the cause.
Error Code: 0
Call: INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:430)
Caused by: javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 4 INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (18, 403) was aborted.  Call getNextException to see the cause.
Error Code: 0
Call: INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
        at org.apache.ambari.server.state.cluster.ClusterImpl.addDesiredConfig(ClusterImpl.java:2180)
        at org.apache.ambari.server.state.cluster.ClusterImpl.addDesiredConfig(ClusterImpl.java:2148)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.updateConfigurationPropertiesForCluster(AbstractUpgradeCatalog.java:598)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.updateConfigurationPropertiesWithValuesFromXml(AbstractUpgradeCatalog.java:521)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.updateConfigurationPropertiesWithValuesFromXml(AbstractUpgradeCatalog.java:482)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addNewConfigurationsFromXml(AbstractUpgradeCatalog.java:421)
        at org.apache.ambari.server.upgrade.UpgradeCatalog242.executeDMLUpdates(UpgradeCatalog242.java:127)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:943)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
        ... 1 more
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 4 INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (18, 403) was aborted.  Call getNextException to see the cause.
Error Code: 0
Call: INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeJDK12BatchStatement(DatabaseAccessor.java:926)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:179)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
        ... 11 more
Caused by: java.sql.BatchUpdateException: Batch entry 4 INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (18, 403) was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2740)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2889)
        at org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.executeBatch(DatabasePlatform.java:2336)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeJDK12BatchStatement(DatabaseAccessor.java:922)
        ... 21 more

ambari-eclipselink.log

[EL Info]: 2016-07-28 09:22:30.365--UnitOfWork(820730256)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
Error Code: 0
Call: SELECT DISTINCT task_id FROM host_role_command WHERE ((role = ?) AND (status = ?)) ORDER BY task_id
	bind => [2 parameters bound]
Query: ReportQuery(referenceClass=HostRoleCommandEntity sql="SELECT DISTINCT task_id FROM host_role_command WHERE ((role = ?) AND (status = ?)) ORDER BY task_id").
[EL Error]: 2016-07-28 09:22:30.372--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:30.372--UnitOfWork(820730256)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReportQuery(referenceClass=HostRoleCommandEntity sql="SELECT DISTINCT task_id FROM host_role_command WHERE ((role = ?) AND (status = ?)) ORDER BY task_id").
[EL Error]: 2016-07-28 09:22:30.831--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:30.833--UnitOfWork(1847253395)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReportQuery(name="HostRoleCommandEntity.findCountByCommandStatuses" referenceClass=HostRoleCommandEntity sql="SELECT COUNT(task_id) FROM host_role_command WHERE (status IN ?)").
[EL Error]: 2016-07-28 09:22:30.834--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:30.834--UnitOfWork(1847253395)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReportQuery(name="HostRoleCommandEntity.findCountByCommandStatuses" referenceClass=HostRoleCommandEntity sql="SELECT COUNT(task_id) FROM host_role_command WHERE (status IN ?)").
[EL Error]: 2016-07-28 09:22:30.913--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:30.915--UnitOfWork(691671603)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReportQuery(referenceClass=RequestEntity sql="SELECT request_id AS a1 FROM request ORDER BY request_id DESC LIMIT ? OFFSET ?").
[EL Error]: 2016-07-28 09:22:30.916--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:30.916--UnitOfWork(691671603)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReportQuery(referenceClass=RequestEntity sql="SELECT request_id AS a1 FROM request ORDER BY request_id DESC LIMIT ? OFFSET ?").
[EL Error]: 2016-07-28 09:22:31.136--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:31.138--UnitOfWork(831902125)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReadAllQuery(name="AlertCurrentEntity.findByHostAndName" referenceClass=AlertCurrentEntity sql="SELECT t1.alert_id AS a1, t1.definition_id AS a2, t1.history_id AS a3, t1.latest_text AS a4, t1.latest_timestamp AS a5, t1.maintenance_state AS a6, t1.original_timestamp AS a7 FROM alert_history t0, alert_definition t2, alert_current t1 WHERE ((((t0.cluster_id = ?) AND (t2.definition_name = ?)) AND (t0.host_name = ?)) AND ((t0.alert_id = t1.history_id) AND (t2.definition_id = t0.alert_definition_id))) LIMIT ? OFFSET ?").
[EL Error]: 2016-07-28 09:22:31.14--ServerSession(879319843)-- 
[EL Info]: 2016-07-28 09:22:31.14--UnitOfWork(831902125)-- Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Error Code: 0
Query: ReadAllQuery(name="AlertCurrentEntity.findByHostAndName" referenceClass=AlertCurrentEntity sql="SELECT t1.alert_id AS a1, t1.definition_id AS a2, t1.history_id AS a3, t1.latest_text AS a4, t1.latest_timestamp AS a5, t1.maintenance_state AS a6, t1.original_timestamp AS a7 FROM alert_history t0, alert_definition t2, alert_current t1 WHERE ((((t0.cluster_id = ?) AND (t2.definition_name = ?)) AND (t0.host_name = ?)) AND ((t0.alert_id = t1.history_id) AND (t2.definition_id = t0.alert_definition_id))) LIMIT ? OFFSET ?").
[EL Warning]: 2016-07-28 09:22:33.401--UnitOfWork(691671603)-- 
[EL Warning]: 2016-07-28 09:58:01.779--ServerSession(879319843)-- The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided.
[EL Info]: 2016-07-28 09:58:03.095--ServerSession(879319843)-- EclipseLink, version: Eclipse Persistence Services - 2.6.2.v20151217-774c696
[EL Info]: 2016-07-28 09:58:03.525--ServerSession(879319843)-- /file:/usr/lib/ambari-server/ambari-server-2.2.2.0.460.jar_ambari-server_url=jdbc:postgresql://localhost/ambari_user=ambari login successful
[EL Warning]: 2016-08-24 16:52:09.931--ServerSession(687329219)-- The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided.
[EL Info]: 2016-08-24 16:52:10.135--ServerSession(687329219)-- EclipseLink, version: Eclipse Persistence Services - 2.6.2.v20151217-774c696
[EL Info]: 2016-08-24 16:52:10.287--ServerSession(687329219)-- /file:/usr/lib/ambari-server/ambari-server-2.2.2.0.460.jar_ambari-views_url=jdbc:postgresql://localhost/ambari_user=ambari login successful
HDP-2.4.2.0 
Ambari 2.4.0.1 
Postgres 8.4.20
2 REPLIES 2
Highlighted

Re: Ambari 2.5.0 upgrade failed

Super Mentor

@Alena Melnikova

The SQL queries are failing. But i think it might be the side effect of inconsistent connection to the Postgresql Database because your log indicates :

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

.

Sometimes due to DB connectivity inconsistency the queries are not executed properly and it affects the unit of work)

So if you have DB dump prior to upgrade then restore your DB and then try to upgrade freshly.

Re: Ambari 2.5.0 upgrade failed

Rising Star

Hi @Jay SenSharma

I have dumps before any upgrades (Ambari 2.2.2.0) and after downgrade (Ambari 2.4.1.0). Dump after downgrade is bad because of Downgrade Can Create Multiple Mappings For Latest Configs Anyway I tried to restore from it and run ambari-server upgrade. The errors the same.

Actually I think that problem with tables clusterconfigmapping, serviceconfigmapping. Is there any way to get correct versions of these tables?

Call: INSERT INTO serviceconfigmapping (config_id, service_config_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:430)
Caused by: javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Don't have an account?
Coming from Hortonworks? Activate your account here