Ambari-server abend on startup - inconsitent error messages

[#1]

Hi All - I am trying to install HDP through Ambari on a brand-new cluster (runnng Ubuntu) and am getting abends when I try to start the ambari-server. I get the following messages:

No errors were found. ERROR: Exiting with exit code 1. REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information.

When I check the ambari-server.log I find:

09 Mar 2017 16:09:52,356 ERROR [main] DBAccessorImpl:109 - Error while creating database accessor com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: ** BEGIN NESTED EXCEPTION ** java.net.ConnectException MESSAGE: Connection refused STACKTRACE: java.net.ConnectException: Connection refused (large number of subsequent traceback lines omitted)

From the ambari-server-database log I get:

Ambari database consistency check started... java.net.ConnectException MESSAGE: Connection refused STACKTRACE: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:244) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271) at com.mysql.jdbc.Connection.createNewIO(Connection.java:2771)

Anybody got any ideas why this might be happening???? The DB that I am targetting is MySQL - it's an install that works ok, that I can get into via the CLI and use normally, the ports seem to be open (I can telnet into it), JDBC is installed, and so forth. So, the DB seems to be functioning fine. I am both baffled and needing help from the community.... any suggestions / help welcomed !

Rick

[#2]

@Rick Turner

In ambari-server.log you will find complete stackTrace of the error can you please share it. The "Caused By.." Section of the error can give us more details.

[#3]

Thanks Jay, but problem solved. See below.

[#4 · Accepted solution]

The logs indicate that the port on the host for MySQL isn't open. Your CLI tests indicate it is. One of them has to be wrong 🙂

Can you do a

grep jdbc /etc/ambari-server/conf/ambari.properties 

and see if the DB properties look correct?

[#5]

Hey Jonathan - thanks for the reply. You put your finger on something that had not occurred to me - your answer was correct - the port was both open and closed at the same time. Turns out to have been a DNS problem - somebody had updated a local DNS with a dud record. Those machines that then used that DNS to resolve hostnames etc got the (wrong) address of a machine that did not have 3306 open. So, they failed - including the Ambari server machine. Machines that happen to have been up for a while had the correct info cached locally, so could connect. Problem solved, and a mystery no more. I must confess that it had me tearing my hair out for a while.... Rick