question Re: Ambari admin stack versions hangs in Support Questions

Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 18:02:14 GMT

We recently upgraded HDP from 2.3 to 2.5 (1245) Ambari version 2.4.1.0.

Also today we tried install zeppelin, which failed.

Probably some data in database is now inconsistent

Ambari server log error is in attachment

Re: Ambari admin stack versions hangs

sshimpi — Tue, 03 Jan 2017 18:04:13 GMT

@Indrek Mäestu

Can you try rnabling developer tools from browser and check the url its hitting/hang asking for?

Also i do not see attachment. Can you please verify once.

Re: Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 18:07:09 GMT

ambari-serverlog.txt

slider-client-installlog.txt

Re: Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 18:20:04 GMT

should be visible all logs now.

also browser dev tools output screenshot

ambari-stacks.png

Re: Ambari admin stack versions hangs

jaimin — Tue, 03 Jan 2017 18:22:23 GMT

@Indrek Mäestu

In addition to providing information asked above by @Sagar Shimpi, can you please check the output for stack version API call. Example stated below:

{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions?ClusterStackVersions/version=2.5&fields=ClusterStackVersions/state",
  "items" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions/1",
      "ClusterStackVersions" : {
        "cluster_name" : "c1",
        "id" : 1,
        "repository_version" : 1,
        "stack" : "HDP",
        "state" : "CURRENT",
        "version" : "2.5"
      }
    }
  ]
}

Please verify that the state should be CURRENT. If it is not then some host is yet not publishing it's version as HDP-2.5 which can cause this issue of Admin stack version to hang

If that's the case then you can identify the hosts that are not publishing their versions as 2.5 by following API call:

http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions?ClusterStackVersions/version=2.5&fields=ClusterStackVersions/*

if this is not the case then we need to look at browser's devtool console for any JS error (highlights in red color) and network tab to debug this issue further.

Re: Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 18:34:56 GMT

it error 500 and ambari server log contains same errors (see attachment above)

curl -u admin -H 'X-Requested-By:admin' -X GET 'http://localhost:8080/api/v1/clusters/Prelive/stack_versions?ClusterStackVersions/version=2.5&fields=ClusterStackVersions/state'

{

"status": 500,

"message": "Server Error"

}

Re: Ambari admin stack versions hangs

jaimin — Tue, 03 Jan 2017 18:51:24 GMT

As per the error in the attached ambari-server log, it seems to be the case of multiple registered repo versions with same stack name and version. can you verify if thats indeed the case by running following API and checking that there are more than one entry with same stack name (HDP) and version (2.5).

http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/stack_versions

please comment the output of above API. If it also gives 500 server error then please check in ambari database for following columns of repo_version table if there are any duplicates for composite of stack_id and version.This can be done by executing below query:

select repo_version_id,stack_id,version,display_name from repo_version;

Re: Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 19:57:41 GMT

Yes error 500 in API output

curl -u admin -H 'X-Requested-By:admin' -X GET 'http://localhost:8080/api/v1/clusters/Prelive/stack_versions'

{

"status": 500,

"message": "Server Error"

}

sql query output in here ambari-repo-version.png

Is it safe to delete 2nd row from database (repo_version_id 201)?

Re: Ambari admin stack versions hangs

indrek_maestu — Tue, 03 Jan 2017 20:24:08 GMT

it isn't safe to delete, because at least cluster_version and host_version tables referencing to repo_version_id.

I don't know how this situation can happen anyway (see ambari-repo-version.png) where data in table repo_version don't match

Re: Ambari admin stack versions hangs

jaimin — Wed, 04 Jan 2017 04:05:54 GMT

@Indrek Mäestu

Please take a backup of your db before doing any further database operations and also confirm that the cluster is on 2.5.0.0-1245 version for HDP-2.5.0.0 stack. This can be done by executing following query and getting same result as printed below:

ambari=> select cluster_version.state,repo_version.version,repo_version.display_name from cluster_version full outer join repo_version on cluster_version.repo_version_id=repo_version.repo_version_id  where cluster_version.state='CURRENT' ORDER BY repo_version.version;
  state  |   version    | display_name
---------+--------------+--------------
 CURRENT | 2.5.0.0-1245 | HDP-2.5.0.0
(1 row)

Also you can check same for host version table to see that all hosts are in 'CURRENT' state and on version 2.5.0.0-1245 for HDP-2.5.0.0 stack.

ambari=> select host_version.host_id,host_version.state,repo_version.version,repo_version.display_name from host_version full outer join repo_version on host_version.repo_version_id=repo_version.repo_version_id  ORDER BY repo_version.version;
 host_id |  state  |   version    | display_name
---------+---------+--------------+--------------
       1 | CURRENT | 2.5.0.0-1245 | HDP-2.5.0.0
(1 row)

I believe it should be safe to do cacading delete operation if all hosts and the cluster are on the latest version. But it will be good if Nate Cole who has more context to this part of ambari code can confirm and comment that deleting a row is the best course of action that can be done from here.

cc @Nate

Re: Ambari admin stack versions hangs

ncole — Wed, 04 Jan 2017 04:24:14 GMT

As @jaimin indicated, it is safe to cascade delete the offending record. At the end of the day, the repository_version_id values should be singular across the cluster_version and host_version tables. Since you will have two records with the same version string, the id-lookup is what matters.

Re: Ambari admin stack versions hangs

indrek_maestu — Wed, 04 Jan 2017 16:09:29 GMT

I found that there was problem in repo_version table, version column value should be "2.5.0.0-1245". I had only "2.5.0.0"

Beacuse of that all services failed start or restart, with this error:

ambari-python-wrap /usr/bin/conf-select set-conf-dir --package hadoop --stack-version 2.5.0.0 --conf-version 0' returned 1. 2.5.0.0 Incorrect stack version

I still don't know how this situation happend.

I updated version column to "2.5.0.0-1245" in database repo_version table