Ambar-server start failes with "DB configs consistency check failed"

[#1]

Recently we have been dealing with some server issues I believe we may have lost power at some point. Upon starting the ambari-server I am met with the "DB configs consistency check failed. Run "ambari-server start --skip-database-check" to skip" message. The message from /var/log/ambari-server/ambari-server-check-database.log is:

2016-11-30 09:25:53,306 INFO - ******************************* Check database started *******************************

2016-11-30 09:25:57,570 INFO - Checking for configs not mapped to any cluster

2016-11-30 09:25:57,593 INFO - Checking for configs selected more than once

2016-11-30 09:25:57,595 INFO - Checking for hosts without state

2016-11-30 09:25:57,596 INFO - Checking host component states count equals host component desired states count

2016-11-30 09:25:57,598 INFO - Checking services and their configs

2016-11-30 09:26:02,972 ERROR - You have non selected configs: ams-hbase-log4j,ams-hbase-security-site,ams-hbase-policy,ams-log4j for service AMBARI_METRICS from cluster slush!

2016-11-30 09:26:02,973 ERROR - You have non selected configs: zeppelin-config,zeppelin-ambari-config for service ZEPPELIN from cluster slush!

2016-11-30 09:26:02,973 INFO - ******************************* Check database completed *******************************

2016-11-30 09:26:12,880 INFO - Checking DB store version

2016-11-30 09:26:13,360 INFO - DB store version is compatible

I have started ambari-server with the "--skip-database-check" option however upon start up no services are show to be in the cluster. I have tried re-adding the services but end up with a server error and stopping at "Preparing to deploy: 18 of 18 tasks completed.

Any advice or path forward would be appreciated.

[#2 · Accepted solution]

@Brian Ramsel

Did you remove ZEPPELIN and AMBARI_METRICS from Ambari in past by any chance?

If that is the case, some traces might have left in Ambari DB.

Can you please follow below steps:

1. Take backup of Ambari DB to be on safer side.

2. Login to Ambari DB, check below tables and remove any leftover traces of AMBARI_METRICS and ZEPPELIN

hostcomponentstate
hostcomponentdesiredstate
servicecomponentdesiredstate
servicedesiredstate
serviceconfighosts
serviceconfigmapping
serviceconfig
requestresourcefilter
requestoperationlevel
clusterservices
clusterconfig

Hope this information helps! 🙂

[#3]

Thank you. This worked out just need a little sql know how to handle the serviceconfigmapping. Thanks again for the quick response!

[#4]

This is what I ran to clean up the issue on my ambari database, thank you for the post:

delete from ambari.serviceconfigmapping where service_config_id in (select service_config_id from ambari.serviceconfig where service_name in ('PXF','HAWQ','ZEPPELIN'));

delete from ambari.hostcomponentstate where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.hostcomponentdesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.servicecomponentdesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.servicedesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.serviceconfig where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.requestresourcefilter where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.requestoperationlevel where service_name in ('PXF','HAWQ','ZEPPELIN');

delete from ambari.clusterservices where service_name in ('PXF','HAWQ','ZEPPELIN');