I have ambari controlling 5 hosts. I can see that although service is running ambari is showing that service is down. How to resolve this. I have stopped and started agent, ambari-server and event restarted the whole server.
Pl check the Database. I believe the Ambari queues are overloaded and stilling waiting for eariler actions response, which results the database is not getting updated.
I have experienced similar issues in the past and it was due to permissions on a PID file. Take a look at the service descriptor files to see what files it checks to see if it's running. Typically it reads a pid and checks if the process listed in the pid file is running. I would discover the location of the pid file it checks then stop the service, delete the pid files, then use ambari to restart the services. This should recreate the pid files with the correct user/group and permissions and ultimately fix the issue you are seeing.PID file location would be /var/run/.
And mostly I have seen this issue beaucse of ambari-agent PID.
you can get the PID of service with :
netstat -nlp | grep portNumber
ps -edf | grep serviceName
and replace the contents of file /var/run/service/service.pid by the PID that who got you back