Ambari server startup failed during upgrade

[#1]

SYMPTOM: While performing HDP upgrade [HDP-2.3.0 upgrade to HDP-2.5.3] there was error for which we had to start ambari server. But ambari server startup failed with below error -

ERROR:

Caused by: Exception [EclipseLink-116] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException 

Exception Description: No conversion value provided for the value [UPGRADED] in field [host_version.state]. 

Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[state-->host_version.state] 

Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostVersionEntity --> [DatabaseTable(host_version)]) 

at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066) 

at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226) 

at org.eclipse.persistence.mappings.converters.EnumTypeConverter.convertDataValueToObjectValue(EnumTypeConverter.java:141) 

at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.getObjectValue(AbstractDirectMapping.java:616) 

at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.valueFromRow(AbstractDirectMapping.java:1220) 

at org.eclipse.persistence.mappings.DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1539) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:462) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1005) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735) 

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689) 

at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805) 

at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:962) 

at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:573) 

at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175) 

at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904) 

at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)

ROOT CAUSE: There was manual entry in ambari db table "host_version" where the 'state' was modified to "UPGRADED" as shown below which cause the issue-

ambari=> select * from host_version;
 id | repo_version_id | host_id |  state  
----+-----------------+---------+---------
  1 |               1 |       1 | CURRENT
  2 |               1 |       3 | CURRENT
  3 |               2 |       2 | UPGRADED

RESOLUTION: Steps followed for resolution -

1. Took ambari DB backup

2. Check the latest repo_version_id using below command -

select (repo_version_id, version) from repo_version ;
    row      
---------------
(1,  2.3.0.1-15)
(2,  2.3.5.3-8)

3. Check for latest version and repo_version_id from above command and modified respective repo_version_id in "host_version" table pointing the status to CURRENT and made rest of the entries to INSTALLED. So the output should look like below -

ambari=> select * from host_version;
 id | repo_version_id | host_id |  state  
----+-----------------+---------+---------
  1 |               1 |       1 | INSTALLED
  2 |               1 |       3 | INSTALLED
  3 |               2 |       2 | CURRENT

4. Restart ambari-server.