Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ambari add new service stuck

Solved Go to solution
Highlighted

ambari add new service stuck

Contributor

Hi, I just installed a new HDP 2.5.3.0 cluster with 10 nodes, when I tried to 'add new service' it is in 'Loading....' and get no where.

this is first time to my cluster, I have installed so many, never have this problem. I did all ambari-server restart, stop all service and start all service, also, chown root:root /var/run/ambari still not let me add new service.

anyone have any ideas to help me here? thank you very much.

Robin

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: ambari add new service stuck

Super Mentor

@Robin Dong

Looks like one of your host is causing the problem.

Please check the following:

1). Check the "host_version" table and findout which host is not pointing to "2.5.3.0-37"

SELECT * FROM host_version;

2). Login to the problematic host using SSH and then run the following command to find out if it really has the 2.5.3.0-37 packages installed or not?

# hdp-select

3). If you see that the hdp-select command output shows that all the packages installed to that host is having the correct version then following the next step.

4). If you find the problematic host then find it's host_id from the "hosts" table and then check if it is pointing to OLD repo_version_id , if yes then update it to point to the current version.

 UPDATE host_version SET state='CURRENT' WHERE host_id='xx';

.

Also you can update the "repo_version_id" for this host to point to the latest version of repo (if it is not set already)

Please make sure that after this change the host is pointing to the repo_version_id mentioned in the "cluster_version" table which has state as "CURRENT". .

5). After the DB changes (commits) ambari-server need to be restarted.

# ambari-server restart

.

View solution in original post

13 REPLIES 13
Highlighted

Re: ambari add new service stuck

Super Mentor

@Robin Dong

Did you notice any warning/error in ambari-server.log? (before or after the UI hung).

The OS where the Ambari is running has enough memory (free -m). Also does ambari has enough Free space left. Following command will list the ambari memory details in run time.

# $JAVA_HOME/bin/jmap -heap  $AMBARI_PID

Did you try opening the ambari UI in incognito mode browser?

.

Highlighted

Re: ambari add new service stuck

Contributor

thank you so much for your respond.

ambari-server.log showed this error: error=/var/run/ambari-server/stack-recommendations/1/stackadvisor.err

[root@ip-172-31-6-79 bin]# free -m

total used free shared buff/cache available

Mem: 7565 1478 119 48 5967 5714

Swap: 0 0 0

Highlighted

Re: ambari add new service stuck

Super Mentor

@Robin Dong

Sometimes this Add Service UI can get hung when some of the Hosts of the cluster are in OUT_OF_SYNC state.

Please check the ambari DB to findout those host states.

SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";

SELECT host_id, host_name FROM hosts where host_id IN (SELECT host_id FROM host_version WHERE state = "OUT_OF_SYNC");

If you find any problematic host then please Enable the "Maintenance Mode" ON for those hosts first and then perform Add Service.

.

Re: ambari add new service stuck

Contributor

Please refer to this image of the current version 6/7, seemed one of my host has issues, but how to fix it?

Highlighted

Re: ambari add new service stuck

Super Mentor

@Robin Dong Please attache the images.

Highlighted

Re: ambari add new service stuck

Contributor

703-version.png

sorry, somehow it not uploaded last time. here it comes. please see current 6/7, one of my host do have issues, but I could not find any alert in any hosts in the cluster.

I dont know how to query on ambari database, I did hdp-select, it is hard to compare the versions on each states on all hosts.

Highlighted

Re: ambari add new service stuck

Contributor

703-version.png

please see the 'current 6/7', it means one of my hosts have issues, but I could not find any alert in my cluster and only warning in deploy.

Highlighted

Re: ambari add new service stuck

Contributor

I saw your ealier post

# mysql -u ambari ambari -pbigdata 

mysql> SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";
[root@ip-172-31-6-79 bin]# mysql
bash: mysql: command not found


Highlighted

Re: ambari add new service stuck

Super Mentor

@Robin Dong

You will need to have "mysql" client packages installed on the host where you are running this command:

For example:

# yum install mysql-community-client

And then you can run the following command:

# mysql -u ambari ambari -pbigdata

.

Don't have an account?
Coming from Hortonworks? Activate your account here