Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

dineshc — Wed, 06 Sep 2017 08:52:41 GMT

Upgrading Ambari 2.5.0.3 to Ambari 2.5.2.0 it fails with following stack trace:

Ambari Logs:

05 Sep 2017 16:58:54,928 ERROR [main] AlertDefinitionFactory:199 - Unable to deserialize the alert definition source during coercion

com.google.gson.JsonSyntaxException: Expecting number, got: STRING
at com.google.gson.internal.bind.TypeAdapters$11.read(TypeAdapters.java:304)
at com.google.gson.internal.bind.TypeAdapters$11.read(TypeAdapters.java:293)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:795)
at com.google.gson.Gson.fromJson(Gson.java:859)
at com.google.gson.Gson$2.deserialize(Gson.java:131)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory$AlertDefinitionSourceAdapter.deserialize(AlertDefinitionFactory.java:373)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory$AlertDefinitionSourceAdapter.deserialize(AlertDefinitionFactory.java:313)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.Gson.fromJson(Gson.java:795)
at com.google.gson.Gson.fromJson(Gson.java:761)
at com.google.gson.Gson.fromJson(Gson.java:710)
at com.google.gson.Gson.fromJson(Gson.java:682)
at org.apache.ambari.server.state.alert.AlertDefinitionFactory.coerce(AlertDefinitionFactory.java:196)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1164)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
05 Sep 2017 16:58:54,929 DEBUG [main] AmbariMetaInfo:1172 - The alert named yarn_app_timeline_server_webui has been modified from the stack definition and will not be merged
05 Sep 2017 16:58:54,929 ERROR [main] SchemaUpgradeHelper:242 - Upgrade failed.
java.lang.NullPointerException
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1177)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
05 Sep 2017 16:58:54,929 ERROR [main] SchemaUpgradeHelper:446 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:243)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:433)
Caused by: java.lang.NullPointerException
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1177)
at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:240)
===================================================================================

On investigation, I found the columns in alert_definition table in ambari database were ordered differently in 2.5.2.0 and 2.5.0.3

I made changes to the table schema and corrected the column order to match the schema in 2.5.2.0, however, I still get the same error.

Appreciate any hints.

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

jonathanhurley — Wed, 06 Sep 2017 20:56:26 GMT

The JSON of the alert definition must have an invalid property set on it. Not sure how it happened, but there are a couple of options:

Find the offending alert definition and correct its JSON in the database. Chances are it's the one after yarn_app_timeline_server_webui.
Just remove all alert data and have Ambari bootstrap the definitions back in from the stack. This is a good option if you haven't added or modified any alert definitions from the ones that ship with the product.

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

dineshc — Thu, 07 Sep 2017 04:24:23 GMT

@Jonathan Hurley thanks for the inputs! Solved it the hard way with help from Amar but you are spot on with your guidance here!

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

jonathanhurley — Thu, 07 Sep 2017 06:40:30 GMT

@Dinesh Chitlangia Glad you got it resolved. By any chance, did you determine which alert definition it was and what the invalid JSON was? We might want to file a bug if we can figure out how it happened.

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

apappu — Thu, 07 Sep 2017 10:37:02 GMT

@Jonathan Hurley it is yarn_app_timeline_server_webui alert. for some reason alert_source had incorrect value.

question Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING in Archives of Support Questions (Read Only)

Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING

Re: Ambari Upgrade Failure : AlertDefinitionFactory - Unable to deserialize the alert definition source during coercion - Expected number, got : STRING