SYMPTOM: While performing HDP upgrade [HDP-2.3.0 upgrade to HDP-2.5.3] there was error for which we had to start ambari server. But ambari server startup failed with below error - ERROR: Caused by: Exception [EclipseLink-116] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException Exception Description: No conversion value provided for the value [UPGRADED] in field [host_version.state]. Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[state-->host_version.state] Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostVersionEntity --> [DatabaseTable(host_version)]) at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066) at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226) at org.eclipse.persistence.mappings.converters.EnumTypeConverter.convertDataValueToObjectValue(EnumTypeConverter.java:141) at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.getObjectValue(AbstractDirectMapping.java:616) at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.valueFromRow(AbstractDirectMapping.java:1220) at org.eclipse.persistence.mappings.DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1539) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:462) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1005) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689) at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805) at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:962) at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:573) at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175) at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904) at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134) ROOT CAUSE: There was manual entry in ambari db table "host_version" where the 'state' was modified to "UPGRADED" as shown below which cause the issue- ambari=> select * from host_version; id | repo_version_id | host_id | state ----+-----------------+---------+--------- 1 | 1 | 1 | CURRENT 2 | 1 | 3 | CURRENT 3 | 2 | 2 | UPGRADED RESOLUTION: Steps followed for resolution - 1. Took ambari DB backup 2. Check the latest repo_version_id using below command - select (repo_version_id, version) from repo_version ; row --------------- (1, 2.3.0.1-15) (2, 2.3.5.3-8) 3. Check for latest version and repo_version_id from above command and modified respective repo_version_id in "host_version" table pointing the status to CURRENT and made rest of the entries to INSTALLED. So the output should look like below - ambari=> select * from host_version; id | repo_version_id | host_id | state ----+-----------------+---------+--------- 1 | 1 | 1 | INSTALLED 2 | 1 | 3 | INSTALLED 3 | 2 | 2 | CURRENT 4. Restart ambari-server. ... View more

SYMPTOM: During upgrade from HDP 2.4.3 to HDP2.6.0 the express upgrade failed on the step "Yarn NodeMangers / Restarting NodeManger." ERROR: 2017-06-22 15:02:22,791 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZK operation. org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /rmstore at org.apache.zookeeper.KeeperException.create(KeeperException.java:123) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ROOT CAUSE: The issue was with property Ambari-Yarn-Configs-"yarn-resourcemanager-zk-address" was set to localhost:2181 RESOLUTION: Modified address to "{zk-server-hostname}:2181" resolved the issue. After which node managers were able to start successfully. ... View more

SYMPTOM: HDP upgrade failed on "Finalize". Tried setting the version manually using below command which was failing - ambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0 ERROR: Please find error screenshot below - ROOT CAUSE: There was issue with the last "request_id" which was run by Express upgrade process. you can check the last request_id from ambari DB from table name "host_role_command" RESOLUTION: 1. Backup the ambari DB 2. Find the last "request_id" from table "host_role_command" and check the status of the request_id. 3. In my case the status was "Failed" 4. Modifying the status of the request_id from "FAILED" to "COMPLETED" and restart ambari server. 5. Try running the command again ambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0 ... View more