Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

Hortonworks,

I have 2 installations of HDP 2.6.3 - One installation on a Centos 7 Virtual Machine, and one installation on a CentOS 7 Linux PC.

Both are configured with different static IP addresses and both were set up identically with MySQL 5.7 for Ambari and Hive.

The VM installation always works every single time on boot. Flawlessly. No issues with Stale PIDs, etc. It just works.

HOWEVER, the PC installation of HDP NEVER starts successfully on boot, despite being configured within Ambari to start all services on boot. For example, I always get the same message:

# ambari-server status

Using python  /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid

After checking the ambari server log at /var/log/ambari-server/ambari-server.log I always see the same error:

BAccessorImpl:119 - Error while creating database accessor
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

There is nothing wrong with the MySQL installation. I can log in and see tables with ambari and hive usernames.

Variations of this same issue "Ambari Server not running, Stale PID" have been posted multiple times throughout this forum. I'm really tired of dealing with this issue with HDP.

PLEASE ADVISE how to fix.

Thank you.

11 REPLIES 11

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

@John Doe

Perhaps you have already checked this but, have you configured sequence/order for services at startup?

If not please review this link

https://www.putorius.net/2016/04/start-specifc-services-first-in-redhat.html

HTH

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

@John Doe

Based on the error you shared I think in bare metal server the network link/mysql database maybe taking longer to startup for some reason.

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Please review this:

https://access.redhat.com/solutions/18015

Most of the diagnostics for this issue are around network/firewall and database.

As I mentioned in previous post perhaps pushing the ambari service start order to perhaps last place may help. Could you explore the above diagnostics and see if changing the order of startup helps in some way?

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

@Felix Albani

Thank you for tyour reply, but that link cannot explain why two different installations configured exactly the same way on the same OS, one works and one doesn't. For example, I didn't have to configure the VM to start services in a different order and it works fine.

This seems to be a major bug in HDP as the question is never really addressed fully in any thread related to this topic.

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

Mentor

@John Doe

Can you re-run this as the process owner of Ambari server, it could look trivial. Also compare the permissions on they mysql-connector -java.jar on both VM and bare metal !

#  ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

You are right if you didn't tweak the startup order on the VM then no reason to do that. After running the above please monitor the ambari-server.log what you could do is do a backup of the old logs for analysis. Could you also compare the ambari.properties of both clusters.

Please revert

HTH

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

Hi @Geoffrey Shelton Okot

Thanks for your reply. I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service.

FYI - I did compare the ambari.properties of both clusters. They are exactly the same. Line for line.

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

Mentor

@John Doe

Thanks for the feedback just monitor and update this thread after re-running the mysql-connector -java.jar.

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

Hi @Geoffrey Shelton Okot

Please see my last reply.

"I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service."

In other words, I have run:

  # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

during setup and after setup.

No changes. I am still encountering Stale PID.

This issue is super frustrating.... :(

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

@Felix AlbaniT

Thank you for your reply, but once again, the boot sequence order has nothing to do with the bare metal dbc4.CommunicationsException: Communications link failure. If it did, the VM would throw the same error since the start sequence is unmodified on that machine, similar to the bare metal instance.

Re: Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

New Contributor

Hi @Felix Albani,

Although I strongly believe that sequence/order for services at startup has nothing to do with the "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure" issue, I am willing to try altering the service order on bare metal IF (and ONLY IF) you can provide clear, step-by-step instructions on how to modify the sequence order within a post on this thread. If you can do this, then I will give it a try. However, please do not include URLs to other sites, as I unfortunately do not have time to research the other links. Again, if you can provide step-by-step instructions on how to modify the sequence order within a post on this thread, I will give it a try.

Please advise. Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here