Upgrading Ambari 2.2 to 2.4 issue

[#1]

Hi, I have upgraded Ambari 2.2 to 2.4 and everything went well as per steps but it failed to start the ambari service and the following is the error

2016-08-31 21:52:28,082 INFO - ******************************* Check database started ******************************* 2016-08-31 21:52:31,647 INFO - Checking for configs not mapped to any cluster 2016-08-31 21:52:31,653 INFO - Checking for configs selected more than once 2016-08-31 21:52:31,655 INFO - Checking for hosts without state 2016-08-31 21:52:31,657 INFO - Checking host component states count equals host component desired states count 2016-08-31 21:52:31,660 INFO - Checking services and their configs 2016-08-31 21:52:33,669 ERROR - Unexpected error, database check failed java.lang.NullPointerException at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkServiceConfigs(DatabaseConsistencyCheckHelper.java:543) at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:115)

Thank you for your help.

thanks

ram

[#2]

@Ramakrishna Pratapa - Can you attach ambari-server-check-database.log file present in /var/log/ambari-server/ directory?

Also please try running:

ambari-server start --skip-database-check

and let us know how it goes

From the first look, the issue appears due to an inconsistency in a service configuration.

Additionally, please paste the output of:

ambari-server --version

-Vivek

[#3]

I was able to start the server using

ambari-server start --skip-database-check

and the following command returns

ambari-server --version

2.4.0.1-1

I will continue my upgrade and see how it goes

Thank you for your help.

Ram

[#4 · Accepted solution]

You've hit a bug. It most likely has to do with a service configuration you still have for a service that's no longer on your stack. I will go ahead and file a Jira to have this fixed in Ambari 2.4.0.1 ... In the meantime, can you run this SQL on your database so we can determine which config is the problem:

SELECT
  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.version
FROM clusterservices cs
JOIN serviceconfig sc
  ON cs.service_name = sc.service_name
  AND cs.cluster_id = sc.cluster_id
JOIN serviceconfigmapping scm
  ON sc.service_config_id = scm.service_config_id
JOIN clusterconfig cc
  ON scm.config_id = cc.config_id
  AND sc.cluster_id = cc.cluster_id
JOIN clusters c
  ON cc.cluster_id = c.cluster_id
  AND sc.stack_id = c.desired_stack_id
WHERE sc.group_id IS NULL
AND sc.service_config_id = (SELECT
  MAX(service_config_id)
FROM serviceconfig sc2
WHERE sc2.service_name = sc.service_name
AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name,
         cs.service_name,
         cc.type_name,
         sc.version

Created AMBARI-18296 to track this...

[#5]

Thank you for the quick response. I thought it may be better to start fresh and manually removed all artifacts related to Ambari 2.2 and Hdp 2.4 and started installing Ambari 2.4.0.1 as fresh install.

I should have waited little longer....

Thank you

Ram

[#6]

how should one use the query results? is this the fix --> delete sc rows where sc.group_id is null. Alternatively if there is a patch(#5621 commit? ) release can you please share the binary download link.

[#7]

You should see if the services reported for each config are still valid. If not, then it could just be an orphaned configuration which needs to be removed from the database.

Ambari 2.4.1 will address this issue by providing a warning but letting the server startup.

[#8]

Note the above query didn't work for us on PostgreSQL 9.2 where Ambari is using the ambari schema instead of public; the following query works in our case:

SELECT
  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.version
FROM ambari.clusterservices cs
JOIN ambari.serviceconfig sc
  ON cs.service_name = sc.service_name
  AND cs.cluster_id = sc.cluster_id
JOIN ambari.serviceconfigmapping scm 
  ON sc.service_config_id = scm.service_config_id
JOIN ambari.clusterconfig cc
  ON scm.config_id = cc.config_id
  AND sc.cluster_id = cc.cluster_id
JOIN ambari.clusters c
  ON cc.cluster_id = c.cluster_id
  AND sc.stack_id = c.desired_stack_id
WHERE sc.group_id IS NULL
AND sc.service_config_id = (SELECT MAX(service_config_id)
  FROM ambari.serviceconfig sc2
  WHERE sc2.service_name = sc.service_name
  AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name,
 cs.service_name,
 cc.type_name,
 sc.version;

[#9]

Hi,

I am getting this same error and I've installed via yum repo: name=ambari-2.4.0.1 - Updates.

Thanks,

Rupert

[#10]

@Ramakrishna Pratapa

Did the upgrade go smoothly? Were you able to upgrade by skipping the consistency check. I am hitting the same problem. Can you please suggest the path forward?