ambari add new service stuck

[#1]

Hi, I just installed a new HDP 2.5.3.0 cluster with 10 nodes, when I tried to 'add new service' it is in 'Loading....' and get no where.

this is first time to my cluster, I have installed so many, never have this problem. I did all ambari-server restart, stop all service and start all service, also, chown root:root /var/run/ambari still not let me add new service.

anyone have any ideas to help me here? thank you very much.

Robin

[#2]

@Robin Dong

Did you notice any warning/error in ambari-server.log? (before or after the UI hung).

The OS where the Ambari is running has enough memory (free -m). Also does ambari has enough Free space left. Following command will list the ambari memory details in run time.

# $JAVA_HOME/bin/jmap -heap  $AMBARI_PID

Did you try opening the ambari UI in incognito mode browser?

.

[#3]

thank you so much for your respond.

ambari-server.log showed this error: error=/var/run/ambari-server/stack-recommendations/1/stackadvisor.err

[root@ip-172-31-6-79 bin]# free -m

total used free shared buff/cache available

Mem: 7565 1478 119 48 5967 5714

Swap: 0 0 0

[#4]

@Robin Dong

Sometimes this Add Service UI can get hung when some of the Hosts of the cluster are in OUT_OF_SYNC state.

Please check the ambari DB to findout those host states.

SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";

SELECT host_id, host_name FROM hosts where host_id IN (SELECT host_id FROM host_version WHERE state = "OUT_OF_SYNC");

If you find any problematic host then please Enable the "Maintenance Mode" ON for those hosts first and then perform Add Service.

.

[#5]

Please refer to this image of the current version 6/7, seemed one of my host has issues, but how to fix it?

[#6]

@Robin Dong Please attache the images.

[#7]

703-version.png

sorry, somehow it not uploaded last time. here it comes. please see current 6/7, one of my host do have issues, but I could not find any alert in any hosts in the cluster.

I dont know how to query on ambari database, I did hdp-select, it is hard to compare the versions on each states on all hosts.

[#8]

703-version.png

please see the 'current 6/7', it means one of my hosts have issues, but I could not find any alert in my cluster and only warning in deploy.

[#9]

I saw your ealier post

# mysql -u ambari ambari -pbigdata 

mysql> SELECT host_id, state FROM host_version WHERE state = "OUT_OF_SYNC";
[root@ip-172-31-6-79 bin]# mysql
bash: mysql: command not found

[#10]

@Robin Dong

You will need to have "mysql" client packages installed on the host where you are running this command:

For example:

# yum install mysql-community-client

And then you can run the following command:

# mysql -u ambari ambari -pbigdata

.