question Upgrading Ambari 2.2 to 2.4 issue in Archives of Support Questions (Read Only)

Upgrading Ambari 2.2 to 2.4 issue

ram_pratapa — Thu, 01 Sep 2016 09:21:24 GMT

Hi, I have upgraded Ambari 2.2 to 2.4 and everything went well as per steps but it failed to start the ambari service and the following is the error

2016-08-31 21:52:28,082 INFO - ******************************* Check database started ******************************* 2016-08-31 21:52:31,647 INFO - Checking for configs not mapped to any cluster 2016-08-31 21:52:31,653 INFO - Checking for configs selected more than once 2016-08-31 21:52:31,655 INFO - Checking for hosts without state 2016-08-31 21:52:31,657 INFO - Checking host component states count equals host component desired states count 2016-08-31 21:52:31,660 INFO - Checking services and their configs 2016-08-31 21:52:33,669 ERROR - Unexpected error, database check failed java.lang.NullPointerException at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkServiceConfigs(DatabaseConsistencyCheckHelper.java:543) at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:115)

Thank you for your help.

thanks

ram

Re: Upgrading Ambari 2.2 to 2.4 issue

vsharma — Thu, 01 Sep 2016 13:24:45 GMT

@Ramakrishna Pratapa - Can you attach ambari-server-check-database.log file present in /var/log/ambari-server/ directory?

Also please try running:

ambari-server start --skip-database-check

and let us know how it goes

From the first look, the issue appears due to an inconsistency in a service configuration.

Additionally, please paste the output of:

ambari-server --version

-Vivek

Re: Upgrading Ambari 2.2 to 2.4 issue

ram_pratapa — Thu, 01 Sep 2016 19:41:12 GMT

I was able to start the server using

ambari-server start --skip-database-check

and the following command returns

ambari-server --version

2.4.0.1-1

I will continue my upgrade and see how it goes

Thank you for your help.

Ram

Re: Upgrading Ambari 2.2 to 2.4 issue

jonathanhurley — Thu, 01 Sep 2016 21:51:41 GMT

You've hit a bug. It most likely has to do with a service configuration you still have for a service that's no longer on your stack. I will go ahead and file a Jira to have this fixed in Ambari 2.4.0.1 ... In the meantime, can you run this SQL on your database so we can determine which config is the problem:

SELECT
  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.version
FROM clusterservices cs
JOIN serviceconfig sc
  ON cs.service_name = sc.service_name
  AND cs.cluster_id = sc.cluster_id
JOIN serviceconfigmapping scm
  ON sc.service_config_id = scm.service_config_id
JOIN clusterconfig cc
  ON scm.config_id = cc.config_id
  AND sc.cluster_id = cc.cluster_id
JOIN clusters c
  ON cc.cluster_id = c.cluster_id
  AND sc.stack_id = c.desired_stack_id
WHERE sc.group_id IS NULL
AND sc.service_config_id = (SELECT
  MAX(service_config_id)
FROM serviceconfig sc2
WHERE sc2.service_name = sc.service_name
AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name,
         cs.service_name,
         cc.type_name,
         sc.version

Created AMBARI-18296 to track this...

Re: Upgrading Ambari 2.2 to 2.4 issue

ram_pratapa — Thu, 01 Sep 2016 22:28:10 GMT

Thank you for the quick response. I thought it may be better to start fresh and manually removed all artifacts related to Ambari 2.2 and Hdp 2.4 and started installing Ambari 2.4.0.1 as fresh install.

I should have waited little longer....

Thank you

Ram

Re: Upgrading Ambari 2.2 to 2.4 issue

rupert_stechman — Wed, 07 Sep 2016 02:43:09 GMT

Hi,

I am getting this same error and I've installed via yum repo: name=ambari-2.4.0.1 - Updates.

Thanks,

Rupert

Re: Upgrading Ambari 2.2 to 2.4 issue

skunkunuru — Thu, 08 Sep 2016 01:13:46 GMT

how should one use the query results? is this the fix --> delete sc rows where sc.group_id is null. Alternatively if there is a patch(#5621 commit? ) release can you please share the binary download link.

Re: Upgrading Ambari 2.2 to 2.4 issue

jonathanhurley — Fri, 09 Sep 2016 02:47:07 GMT

You should see if the services reported for each config are still valid. If not, then it could just be an orphaned configuration which needs to be removed from the database.

Ambari 2.4.1 will address this issue by providing a warning but letting the server startup.

Re: Upgrading Ambari 2.2 to 2.4 issue

demarkle — Fri, 23 Sep 2016 03:16:43 GMT

Note the above query didn't work for us on PostgreSQL 9.2 where Ambari is using the ambari schema instead of public; the following query works in our case:

SELECT
  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.version
FROM ambari.clusterservices cs
JOIN ambari.serviceconfig sc
  ON cs.service_name = sc.service_name
  AND cs.cluster_id = sc.cluster_id
JOIN ambari.serviceconfigmapping scm 
  ON sc.service_config_id = scm.service_config_id
JOIN ambari.clusterconfig cc
  ON scm.config_id = cc.config_id
  AND sc.cluster_id = cc.cluster_id
JOIN ambari.clusters c
  ON cc.cluster_id = c.cluster_id
  AND sc.stack_id = c.desired_stack_id
WHERE sc.group_id IS NULL
AND sc.service_config_id = (SELECT MAX(service_config_id)
  FROM ambari.serviceconfig sc2
  WHERE sc2.service_name = sc.service_name
  AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name,
 cs.service_name,
 cc.type_name,
 sc.version;

Re: Upgrading Ambari 2.2 to 2.4 issue

bipin_alugubell — Wed, 25 Jan 2017 09:51:26 GMT

@Ramakrishna Pratapa

Did the upgrade go smoothly? Were you able to upgrade by skipping the consistency check. I am hitting the same problem. Can you please suggest the path forward?

Re: Upgrading Ambari 2.2 to 2.4 issue

ram_pratapa — Wed, 25 Jan 2017 21:59:33 GMT

Hi, Good morning, I ran into different issues and I felt it may be easy to do fresh install 2.4 after removing the 2.2. So I went with fresh install.

Thanks

Ram

Re: Upgrading Ambari 2.2 to 2.4 issue

bipin_alugubell — Thu, 26 Jan 2017 01:21:14 GMT

@Ramakrishna Pratapa Thank you for your response. I already have a cluster running with 2.2 and now I upgraded to 2.4. Is it possible to remove ambari completely and install newly for an existing cluster? Can you please point me to the steps.

Re: Upgrading Ambari 2.2 to 2.4 issue

ram_pratapa — Thu, 26 Jan 2017 01:28:00 GMT

I don't have any data in the cluster and hence it was easy for me to remove all bits from the nodes and did fresh install with 2.4. But if you have data in the cluster it may be better to proceed with cluster upgrade steps and verify them as you have already upgraded the Ambari to 2.4.

Thanks

Ram