Created 11-30-2016 02:53 PM
Recently we have been dealing with some server issues I believe we may have lost power at some point. Upon starting the ambari-server I am met with the "DB configs consistency check failed. Run "ambari-server start --skip-database-check" to skip" message. The message from /var/log/ambari-server/ambari-server-check-database.log is:
2016-11-30 09:25:53,306 INFO - ******************************* Check database started *******************************
2016-11-30 09:25:57,570 INFO - Checking for configs not mapped to any cluster
2016-11-30 09:25:57,593 INFO - Checking for configs selected more than once
2016-11-30 09:25:57,595 INFO - Checking for hosts without state
2016-11-30 09:25:57,596 INFO - Checking host component states count equals host component desired states count
2016-11-30 09:25:57,598 INFO - Checking services and their configs
2016-11-30 09:26:02,972 ERROR - You have non selected configs: ams-hbase-log4j,ams-hbase-security-site,ams-hbase-policy,ams-log4j for service AMBARI_METRICS from cluster slush!
2016-11-30 09:26:02,973 ERROR - You have non selected configs: zeppelin-config,zeppelin-ambari-config for service ZEPPELIN from cluster slush!
2016-11-30 09:26:02,973 INFO - ******************************* Check database completed *******************************
2016-11-30 09:26:12,880 INFO - Checking DB store version
2016-11-30 09:26:13,360 INFO - DB store version is compatible
I have started ambari-server with the "--skip-database-check" option however upon start up no services are show to be in the cluster. I have tried re-adding the services but end up with a server error and stopping at "Preparing to deploy: 18 of 18 tasks completed.
Any advice or path forward would be appreciated.
Created 11-30-2016 07:22 PM
Did you remove ZEPPELIN and AMBARI_METRICS from Ambari in past by any chance?
If that is the case, some traces might have left in Ambari DB.
Can you please follow below steps:
1. Take backup of Ambari DB to be on safer side.
2. Login to Ambari DB, check below tables and remove any leftover traces of AMBARI_METRICS and ZEPPELIN
hostcomponentstate hostcomponentdesiredstate servicecomponentdesiredstate servicedesiredstate serviceconfighosts serviceconfigmapping serviceconfig requestresourcefilter requestoperationlevel clusterservices clusterconfig
Hope this information helps! 🙂
Created 11-30-2016 07:22 PM
Did you remove ZEPPELIN and AMBARI_METRICS from Ambari in past by any chance?
If that is the case, some traces might have left in Ambari DB.
Can you please follow below steps:
1. Take backup of Ambari DB to be on safer side.
2. Login to Ambari DB, check below tables and remove any leftover traces of AMBARI_METRICS and ZEPPELIN
hostcomponentstate hostcomponentdesiredstate servicecomponentdesiredstate servicedesiredstate serviceconfighosts serviceconfigmapping serviceconfig requestresourcefilter requestoperationlevel clusterservices clusterconfig
Hope this information helps! 🙂
Created 11-30-2016 08:41 PM
Thank you. This worked out just need a little sql know how to handle the serviceconfigmapping. Thanks again for the quick response!
Created 01-30-2017 03:50 PM
This is what I ran to clean up the issue on my ambari database, thank you for the post:
delete from ambari.serviceconfigmapping where service_config_id in (select service_config_id from ambari.serviceconfig where service_name in ('PXF','HAWQ','ZEPPELIN'));
delete from ambari.hostcomponentstate where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.hostcomponentdesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.servicecomponentdesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.servicedesiredstate where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.serviceconfig where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.requestresourcefilter where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.requestoperationlevel where service_name in ('PXF','HAWQ','ZEPPELIN');
delete from ambari.clusterservices where service_name in ('PXF','HAWQ','ZEPPELIN');