Ambari admin stack versions hangs

[#1]

We recently upgraded HDP from 2.3 to 2.5 (1245) Ambari version 2.4.1.0.

Also today we tried install zeppelin, which failed.

Probably some data in database is now inconsistent

Ambari server log error is in attachment

[#2 · Accepted solution]

@Indrek Mäestu

Can you try rnabling developer tools from browser and check the url its hitting/hang asking for?

Also i do not see attachment. Can you please verify once.

[#3]

should be visible all logs now.

also browser dev tools output screenshot

ambari-stacks.png

[#4]

ambari-serverlog.txt

slider-client-installlog.txt

[#5]

@Indrek Mäestu

In addition to providing information asked above by @Sagar Shimpi, can you please check the output for stack version API call. Example stated below:

{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions?ClusterStackVersions/version=2.5&fields=ClusterStackVersions/state",
  "items" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions/1",
      "ClusterStackVersions" : {
        "cluster_name" : "c1",
        "id" : 1,
        "repository_version" : 1,
        "stack" : "HDP",
        "state" : "CURRENT",
        "version" : "2.5"
      }
    }
  ]
}

Please verify that the state should be CURRENT. If it is not then some host is yet not publishing it's version as HDP-2.5 which can cause this issue of Admin stack version to hang

If that's the case then you can identify the hosts that are not publishing their versions as 2.5 by following API call:

http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions?ClusterStackVersions/version=2...

if this is not the case then we need to look at browser's devtool console for any JS error (highlights in red color) and network tab to debug this issue further.

[#6]

it error 500 and ambari server log contains same errors (see attachment above)

curl -u admin -H 'X-Requested-By:admin' -X GET 'http://localhost:8080/api/v1/clusters/Prelive/stack_versions?ClusterStackVersions/version=2.5&fields=ClusterStackVersions/state'

{

"status": 500,

"message": "Server Error"

}

[#7]

As per the error in the attached ambari-server log, it seems to be the case of multiple registered repo versions with same stack name and version. can you verify if thats indeed the case by running following API and checking that there are more than one entry with same stack name (HDP) and version (2.5).

http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions

please comment the output of above API. If it also gives 500 server error then please check in ambari database for following columns of repo_version table if there are any duplicates for composite of stack_id and version.This can be done by executing below query:

select repo_version_id,stack_id,version,display_name from repo_version;

[#8]

Yes error 500 in API output

curl -u admin -H 'X-Requested-By:admin' -X GET 'http://localhost:8080/api/v1/clusters/Prelive/stack_versions'

{

"status": 500,

"message": "Server Error"

}

sql query output in here ambari-repo-version.png

Is it safe to delete 2nd row from database (repo_version_id 201)?

[#9]

it isn't safe to delete, because at least cluster_version and host_version tables referencing to repo_version_id.

I don't know how this situation can happen anyway (see ambari-repo-version.png) where data in table repo_version don't match

[#10]

@Indrek Mäestu

Please take a backup of your db before doing any further database operations and also confirm that the cluster is on 2.5.0.0-1245 version for HDP-2.5.0.0 stack. This can be done by executing following query and getting same result as printed below:

ambari=> select cluster_version.state,repo_version.version,repo_version.display_name from cluster_version full outer join repo_version on cluster_version.repo_version_id=repo_version.repo_version_id  where cluster_version.state='CURRENT' ORDER BY repo_version.version;
  state  |   version    | display_name
---------+--------------+--------------
 CURRENT | 2.5.0.0-1245 | HDP-2.5.0.0
(1 row)

Also you can check same for host version table to see that all hosts are in 'CURRENT' state and on version 2.5.0.0-1245 for HDP-2.5.0.0 stack.

ambari=> select host_version.host_id,host_version.state,repo_version.version,repo_version.display_name from host_version full outer join repo_version on host_version.repo_version_id=repo_version.repo_version_id  ORDER BY repo_version.version;
 host_id |  state  |   version    | display_name
---------+---------+--------------+--------------
       1 | CURRENT | 2.5.0.0-1245 | HDP-2.5.0.0
(1 row)

I believe it should be safe to do cacading delete operation if all hosts and the cluster are on the latest version. But it will be good if Nate Cole who has more context to this part of ambari code can confirm and comment that deleting a row is the best course of action that can be done from here.

cc @Nate