Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

[#1]

Upgrading Ambari 2.5.0.3 to Ambari 2.5.2.0 it fails with following stack trace:

Ambari Logs:

05 Sep 2017 16:58:54,928 ERROR [main] AlertDefinitionFactory:199 - Unable to deserialize the alert definition source during coercion

com.google.gson.JsonSyntaxException: Expecting number, got: STRING
at com.google.gson.internal.bind.TypeAdapters$11.read(TypeAdapters.java:304)
at com.google.gson.internal.bind.TypeAdapters$11.read(TypeAdapters.java:293)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:795)
at com.google.gson.Gson.fromJson(Gson.java:859)
at com.google.gson.Gson$2.deserialize(Gson.java:131)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory$AlertDefinitionSourceAdapter.deserialize(AlertDefinitionFactory.java:373)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory$AlertDefinitionSourceAdapter.deserialize(AlertDefinitionFactory.java:313)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.Gson.fromJson(Gson.java:795)
at com.google.gson.Gson.fromJson(Gson.java:761)
at com.google.gson.Gson.fromJson(Gson.java:710)
at com.google.gson.Gson.fromJson(Gson.java:682)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory.coerce(AlertDefinitionFactory.java:196)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1164)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
05 Sep 2017 16:58:54,929 DEBUG [main] AmbariMetaInfo:1172 - The alert named yarn_app_timeline_server_webui has been modified from the stack definition and will not be merged
05 Sep 2017 16:58:54,929 ERROR [main] SchemaUpgradeHelper:242 - Upgrade failed.
java.lang.NullPointerException
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1177)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
05 Sep 2017 16:58:54,929 ERROR [main] SchemaUpgradeHelper:446 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:243)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
Caused by: java.lang.NullPointerException
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1177)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
===================================================================================

On investigation, I found the columns in alert_definition table in ambari database were ordered differently in 2.5.2.0 and 2.5.0.3

I made changes to the table schema and corrected the column order to match the schema in 2.5.2.0, however, I still get the same error.

Appreciate any hints.

[#2 · Accepted solution]

The JSON of the alert definition must have an invalid property set on it. Not sure how it happened, but there are a couple of options:

• Find the offending alert definition and correct its JSON in the database. Chances are it's the one after yarn_app_timeline_server_webui.
• Just remove all alert data and have Ambari bootstrap the definitions back in from the stack. This is a good option if you haven't added or modified any alert definitions from the ones that ship with the product.

[#3]

@Jonathan Hurley thanks for the inputs! Solved it the hard way with help from Amar but you are spot on with your guidance here!

[#4]

@Dinesh Chitlangia Glad you got it resolved. By any chance, did you determine which alert definition it was and what the invalid JSON was? We might want to file a bug if we can figure out how it happened.

[#5]

@Jonathan Hurley it is yarn_app_timeline_server_webui alert. for some reason alert_source had incorrect value.