Unable to run the ambari_server_stale_alerts alert: java.lang.NullPointerException

[#1]

Hi,

I am using hdp 2.4 with ambari 2.2. I am continuously seeing the following message in ambari server logs.

Unable to run the ambari_server_stale_alerts alert: java.lang.NullPointerException

WARN [qtp-ambari-client-62] servletHandler:563 - /api/v1/clusters/<ournameofCluster>/alerts

The curl command to ambari alerts received status 500 server error.

curl -u <user>:<password> -X GET http://<host>:8080/api/v1/clusters/<nameofourcluster>/alerts

the same curl command is working for alert_definitions.

Please help me on this asap.

Thanks

[#2 · Accepted solution]

@mravipati

Please check the stack Trace of NullPointerException to verify if it is the following API that is causing the NullPointer? This you can find in ambari-server.log

Caught a runtime exception executing a query java.lang.NullPointerException
      at org.apache.ambari.server.controller.internal.AlertResourceProvider.toResource(AlertResourceProvider.java

.

If yes then please check if there is any inconsistency int he following two tables in your Ambari Database:

>>> select * from alert_current where definition_id not in (select definition_id from alert_definition);

>>> select * from alert_history where alert_definition_id not in (select definition_id from alert_definition);

.

If above queries returns any result then please delete those records as following then restart ambari server.

>>> delete from alert_history where alert_definition_id not in (select definition_id from alert_definition); 

>>> delete from alert_current where definition_id not in (select definition_id from alert_definition);  

.

Also the following:

>>> select * from alert_current where history_id not in (select alert_id from alert_history);
>>> delete from alert_current where history_id not in (select alert_id from alert_history);

Then restart ambari-server and login to ambari UI then doublecheck the logs if NPE error is gone.

[#3]

@Jay

Thanks for your immediate reply. The above commands resolved the issue.

[#4]

Those tables have referential integrity and should not be able to get out-of-sync. Are you using MySQL as your Ambari database? If so, is your default engine InnoDB or MyISAM? If it's MyISAM, that's very bad as it doesn't support transactions or foreign keys. You would need perform several steps to convert your database into InnoDB or this could happen again in the future.