HDP 2.5 - Ambari 2.4.0.1 - Ambari can't start/stop/restart services

[#1]

I need help because my sandbox environment are having a strange behavior. I can't start/stop/restart services neither through Ambari neither using the REST API. As the below screenshot shows, after few seconds Ambari get a timedout error.

Also the status of some services changes frequently without send any command. I updated the postgressql database to version 9.4, I reintalled the ambari-agent, delete temporary/cache files of ambari-server and ambari-agent.

I found that always there are some threads between Ambari and the database. Maybe these connections not let Ambari send more commands? I tried to kill all the process, but after a couple of seconds all threads appear again...

Someone have an idea of what is happening to the cluster? If you need more information, request it and I will provide it.

Thanks in advanced.

[#2 · Accepted solution]

I don't think the idle processes are your problem. It is normal to see these when using Postgres. They are related to the connection pool that Ambari uses with Postgres.

[#3]

Also, you may want to take a look at the ambari logs to see if there is anything helpful:

/var/log/ambari-server/ambari-server.log

/var/log/ambari-agent/ambari-agent.log

[#4]

Thanks bhagan.

Below logs are from /var/log/ambari-server/ambari-server.log, are a bit strange and I don't know how interpret it. The configuration of the ambari-agent is the default one and all services are installed in the same server as the ambari-server (all ports are open).

[#5]

Can you make sure that only one ambari-agent is running:

ls -la /var/run/ambari-agent/

There should be one pid file.

[#6]

There is only one ambari-agent running, but I just discovered that every minute the PID change.

I have reviewed again the ambari-agent log and exactly every minute appear the below error (Error in responseId sequence - restarting). So for a reason, every minute the ambari-agent is restarted. If I stop the ambari agent, the service is not start automatically, so the agent is only restarting when is running.

Any idea of what is the root cause of this strange behavior? Is the first time that I see something like this.

Thanks for your time bhagan.

[#7]

It wasn't clear to me from this thread...have you shut down both ambari and the ambari-agent? If not, I would perform the following:

Stop ambari.

Stop the agent.

Start ambari.

Start the agent with the command, ambari-agent start --verbose. Then perhaps include the ambari-agent log as an attachment.

[#8]

Yes, I did that steps.

Here you have an example of the verbose log of Ambari Agent. Each text file is the log generated in each restart of the ambari-agent.

Logs -->ambari-log1.txt and ambari-log2.txt