How to resolve Ambari DB inconsistency - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster

[#1]

PROBLEM:

Ambari Server won't be able to start because of DB inconsistencies.

Sample Error:

2017-02-06 05:08:43,975 ERROR - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster XXXX! 
2017-02-06 05:08:43,976 INFO - ******************************* Check database completed ******************************* 
2017-02-06 05:10:12,834 INFO - Checking DB store version 
2017-02-06 05:10:14,094 INFO - DB store version is compatible 
2017-02-07 13:50:31,769 INFO - ******************************* Check database started ******************************* 
2017-02-07 13:50:41,247 INFO - Checking for configs not mapped to any cluster 
2017-02-07 13:50:41,322 INFO - Checking for configs selected more than once 
2017-02-07 13:50:41,326 INFO - Checking for hosts without state 
2017-02-07 13:50:41,330 INFO - Checking host component states count equals host component desired states count 
2017-02-07 13:50:41,334 INFO - Checking services and their configs 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / SQOOP 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / HDFS 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / MAPREDUCE2 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / TEZ 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / SPARK 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / HBASE 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / ZOOKEEPER 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / YARN 
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / KNOX 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / PIG 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / RANGER 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / HIVE 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / SLIDER 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / AMBARI_INFRA 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / KAFKA 
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / SMARTSENSE 
2017-02-07 13:50:45,809 ERROR - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster XXXXX! 
2017-02-07 13:50:45,810 INFO - ******************************* Check database completed *******************************

.

BUSINESS IMPACT:

It's not recommended to make any changes in service configurations because backend Database is not consistent.

.

WORKAROUND:

ambari-server start --skip-database-check

Note - This is not recommended for production clusters, if you do this, please do not make any modifications in service configurations till you resolves the conflicts.

.

RESOLUTION:

1. Stop Ambari server

ambari-server stop

.

2. Take backup of Ambari Database

For postgres - Use pg_dump command.

For MySql - Use mysqldump command.

.

3. Run below queries to resolve conflicts

delete from hostcomponentstate where service_name = 'ZEPPELIN';

delete from hostcomponentdesiredstate where service_name = 'ZEPPELIN';

delete from servicecomponentdesiredstate where service_name = 'ZEPPELIN';

delete from servicedesiredstate where service_name = 'ZEPPELIN';

delete from serviceconfighosts where service_config_id in (select service_config_id from serviceconfig where service_name = 'ZEPPELIN');

delete from serviceconfigmapping where service_config_id in (select service_config_id from serviceconfig where service_name = 'ZEPPELIN');

delete from serviceconfig where service_name = 'ZEPPELIN';

delete from requestresourcefilter where service_name = 'ZEPPELIN';

delete from requestoperationlevel where service_name = 'ZEPPELIN';

delete from clusterservices where service_name ='ZEPPELIN';

delete from clusterconfig where type_name like 'zeppelin%';

delete from clusterconfigmapping where type_name like 'zeppelin%';

.

4. Start Ambari Server and it should come up without any inconsistencies.

.

Please feel free to comment if you need any further help on this. Happy Hadooping!!