Ambari 2.4 Upgrade failed Database consistency check

[#1]

I have hit the following error while upgrading the ambari and HDP stack

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures}

2017-01-24 15:05:48,420 ERROR - Unexpected error, database check failed

java.lang.NullPointerException

at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkServiceConfigs(DatabaseConsistencyCheckHelper.java:543)

at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:115)

I have started the ambari-server by -skip-database-check.

But, is it good to proceed with the upgrade? What are the steps that I need to follow to continue with the upgrade?

[#2]

@Bipinkumar Algubelli

It is advised to correct Database consistency issues before HDP upgrade. looks like some of the services are having inconsistency. what is the exact Ambari version are you using?

[#3]

@Bipinkumar Algubelli I'd suggest fixing the ambari database inconsistency first prior to upgrade else it may cause issues during upgrade. You can find leads about inconsistency at /var/log/ambari-server/ambari-server-check-database.log from Ambari server.

[#4]

@apappu My previous ambari version was 2.2. I have just upgraded it to 2.4

[#5]

@apatel I am unable to figure out the issue from /var/log/ambari-server/ambari-server-check-database.log

2017-01-24 17:25:35,866 INFO - ******************************* Check database started ******************************* 2017-01-24 17:25:40,576 INFO - Checking for configs not mapped to any cluster 2017-01-24 17:25:40,600 INFO - Checking for configs selected more than once 2017-01-24 17:25:40,603 INFO - Checking for hosts without state 2017-01-24 17:25:40,604 INFO - Checking host component states count equals host component desired states count 2017-01-24 17:25:40,606 INFO - Checking services and their configs 2017-01-24 17:25:42,908 ERROR - Unexpected error, database check failed java.lang.NullPointerException at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkServiceConfigs(DatabaseConsistencyCheckHelper.java:543) at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:115)

[#6]

@apappu @apatel I searched and found that this is a know issue https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-release-notes/content/ambari_reln.... I skipped the database check. But I am worried about the upgrade

[#7]

@Bipinkumar Algubelli

throwing NullPointerexception is known issue - but you will have to find out why it is throwing that exception and have to fix that first.

you can run below query and find out which service it is.

SELECT
  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.version
FROM clusterservices cs
JOIN serviceconfig sc
  ON cs.service_name = sc.service_name
  AND cs.cluster_id = sc.cluster_id
JOIN serviceconfigmapping scm
  ON sc.service_config_id = scm.service_config_id
JOIN clusterconfig cc
  ON scm.config_id = cc.config_id
  AND sc.cluster_id = cc.cluster_id
JOIN clusters c
  ON cc.cluster_id = c.cluster_id
  AND sc.stack_id = c.desired_stack_id
WHERE sc.group_id IS NULL
AND sc.service_config_id = (SELECT
  MAX(service_config_id)
FROM serviceconfig sc2
WHERE sc2.service_name = sc.service_name
AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name,
         cs.service_name,
         cc.type_name,
         sc.version

[#8]

@apappu I tried the query and it gave me 103 rows of result. Should I remove all those configurations?

[#9]

@Bipinkumar Algubelli

I am sorry - please ignore my previous query. can you please upgrade your Ambari to 2.4.2 - then try to start the Ambari server and check /var/log/ambari-server/ambari-server-check-database.log log for error messages. based on the error message you can check hostcomponfrom that you will have to check some of the services in hostcomponentstate table and fix the issues.

[#10]

Bipinkumar Algubelli first try to upgrade ambari version to 2.4.1.0 & then try to upgrade HDP to 2.4. There are manual steps which you can follow to upgrade to 2.4 as well as the docs provided at

https://docs.hortonworks.com/index.html