Support Questions

Find answers, ask questions, and share your expertise

Hi i am getting the following errors while starting the ambari server ... How can i fix this ??

java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:173) at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65) ... 15 more

13 REPLIES 13

Super Mentor

@Arjun Das


As this error seems to be coming while attmepting to connect to MySQL database.

java.net.ConnectException: Connection refused (Connection refused)
.
com.mysql.cj.protocol.StandardSocketFactory.connect


So please check few things:

0. The IPTables/Firewall is disabled on mysql server side so that ambari host can access the MySQL.

# service iptables stop
# sysctl firewalld stop


1. MySQL process is runnign fine and from ambari server host are you able to connect to it?

# telnet $MYSQL_HOST  $MYSQL_PORT
Like:
# telnet mysqlserver.example.com 3306


2. Also please make sure that you are using the correct MySQL JDBC driver JAR (correct version of Driver) and correct mysql Driver classname and the ambari server setip is executed fine as mentioned in:

# ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar


https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_using_amba...

Still if you find any issue then please share the complete stack trace (partial stacktrace will not give much idea) and also please share the ambari.properties.

# grep 'jdbc' /etc/ambari-server/conf/ambari.properties

.

Super Mentor

@Arjun Das

Additionally in your Stacktrace i see the API "com.mysql.cj" being invoked.

You might be using a higher version of mysql-connector-java JDBC Driver.

As the new API "com.mysql.cj" can be seen in the StackTrace.

Please make sure that you are using the correct MySQL JDBC driver. Like if you are using HDP 2.6.5 then MySQL 5.7 Database and respective JDBC driver version needs to be used.

Similar HCC thread: https://community.hortonworks.com/questions/211807/install-mysql-connector-for-hive-metastore.html

grep 'jdbc' /etc/ambari-server/conf/ambari.properties custom.mysql.jdbc.name=mysql-connector-java-5.1.47.jar server.jdbc.connection-pool=c3p0 server.jdbc.connection-pool.acquisition-size=5 server.jdbc.connection-pool.idle-test-interval=7200 server.jdbc.connection-pool.max-age=0 server.jdbc.connection-pool.max-idle-time=14400 server.jdbc.connection-pool.max-idle-time-excess=0 server.jdbc.database=mysql server.jdbc.database_name=ambari_data server.jdbc.driver=com.mysql.jdbc.Driver server.jdbc.driver.path=/usr/lib/java/mysql-connector-java-8.0.12.jar server.jdbc.hostname=localhost server.jdbc.port=3306 server.jdbc.rca.driver=com.mysql.jdbc.Driver server.jdbc.rca.url=jdbc:mysql://N9722:3306/ambari_data server.jdbc.rca.user.name=root server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:mysql://N9722:3306/ambari_data server.jdbc.user.name=root server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

Super Mentor

@Arjun Das

You are using mysql-connector-java JDBC driver of version 8 which is not right.

server.jdbc.driver.path=/usr/lib/java/mysql-connector-java-8.0.12.jar 

Please use mysql 5.6/5.7 JAR.

Please refer to this post to understand this in detail: https://community.hortonworks.com/questions/211807/install-mysql-connector-for-hive-metastore.html

Please make sure that you are using the correct MySQL JDBC driver. Like if you are using HDP 2.6.5 then MySQL 5.7 Database and respective JDBC driver version needs to be used.

MySQL connector 5.1 download link: https://dev.mysql.com/downloads/connector/j/5.1.html

So doenload the correct version of MySQL driver and then put it on ambari server host somewhere and then run the following command to setup the JDBC driver.

# ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar
# ambari-server restart

Please make sure to use the correct driver path based on your setup : /path/to/mysql/mysql-connector-java.jar

.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-administration/content/using_amba...

Hi i am still getting the same error

Waiting for server start.....................................................................ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information.

root@N9722:~# grep 'jdbc' /etc/ambari-server/conf/ambari.properties custom.mysql.jdbc.name=mysql-connector-java-5.1.47.jar server.jdbc.connection-pool=c3p0 server.jdbc.connection-pool.acquisition-size=5 server.jdbc.connection-pool.idle-test-interval=7200 server.jdbc.connection-pool.max-age=0 server.jdbc.connection-pool.max-idle-time=14400 server.jdbc.connection-pool.max-idle-time-excess=0 server.jdbc.database=mysql server.jdbc.database_name=ambari_data server.jdbc.driver=com.mysql.jdbc.Driver server.jdbc.driver.path=/usr/lib/java/mysql-connector-java-5.1.47.jar server.jdbc.hostname=localhost server.jdbc.port=3306 server.jdbc.rca.driver=com.mysql.jdbc.Driver server.jdbc.rca.url=jdbc:mysql://N9722:3306/ambari_data server.jdbc.rca.user.name=root server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:mysql://N9722:3306/ambari_data server.jdbc.user.name=root server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

root@N9722:~# tail -10f /var/log/ambari-server/ambari-server.log at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301) ... 20 more

Super Mentor

@Arjun Das

As mentioned earlier instead of posting just few lines (using tailf) of error it will be really great if you can share the Whole stacktrace.

We can quickly resolve the issue if we have the complete information once.

So can you please share the following details:

1. Why do we see different hostname in your JDBC URL like sometimes it has "localhost" and in some places "N9722" inside ambari.properties?

2. Are you sure that MySQL database is running fine and listening to 3306 port on the correct host?

# netstat -tnlpa | grep 3306

.

Also please check if you are able to connect to the MySQL instance using the telnet?

# telnet localhost 3306
# telnet N9722 3306

3. Are you sure that you want to set the ambari DB owner as "root" ? Or this is by mistake.

server.jdbc.user.name=root

4. Have you already setup the ambari DB as mentioned in the following doc: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-administration/content/using_amba...

Specially the following part:

 # mysql -u root -p 
CREATE USER '<AMBARIUSER>'@'%' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'%';
CREATE USER '<AMBARIUSER>'@'localhost' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'localhost';
CREATE USER '<AMBARIUSER>'@'<AMBARISERVERFQDN>' IDENTIFIED BY '<AMBARIPASSWORD>';
GRANT ALL PRIVILEGES ON *.* TO '<AMBARIUSER>'@'<AMBARISERVERFQDN>';
FLUSH PRIVILEGES

.


at com.mchange.v2.c3p0.DriverManagerDataSource.driver(DriverManagerDataSource.java:285) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

Mentor

@Arjun Das
Can you at least share 200 lines of the /var/log/ambari-server/ambari-server.log in your ambari.properties make sure the server.jdbc.hostname= [FQDN] matches your hostname -f output.

netstat -tnlpa | grep 3306 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:57016 127.0.0.1:3306 ESTABLISHED 2472/mysql-workbenc tcp 0 0 127.0.0.1:3306 127.0.0.1:57018 ESTABLISHED - tcp 0 0 127.0.0.1:3306 127.0.0.1:57016 ESTABLISHED - tcp 0 0 127.0.0.1:57018 127.0.0.1:3306 ESTABLISHED 2472/mysql-workbenc

telnet localhost 3306 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. [ 5.7.23-0ubuntu0.16.04.1 ;8+TQI}o/, bL=Wmysql_native_passwordConnection closed by foreign host.

telnet N9722 3306 Trying 127.0.1.1... telnet: Unable to connect to remote host: Connection refused

yes i want to set server.jdbc.user.name=root

,

netstat -tnlpa | grep 3306 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:57016 127.0.0.1:3306 ESTABLISHED 2472/mysql-workbenc tcp 0 0 127.0.0.1:3306 127.0.0.1:57018 ESTABLISHED - tcp 0 0 127.0.0.1:3306 127.0.0.1:57016 ESTABLISHED - tcp 0 0 127.0.0.1:57018 127.0.0.1:3306 ESTABLISHED 2472/mysql-workbenc

telnet localhost 3306 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. [ 5.7.23-0ubuntu0.16.04.1 ;8+TQI}o/, bL=Wmysql_native_passwordConnection closed by foreign host.

telnet N9722 3306 Trying 127.0.1.1... telnet: Unable to connect to remote host: Connection refused

yes i want to set server.jdbc.user.name=root

Mentor

@Arjun Das

What is the output of the command

$ sudo hostname 

Check the hostname

$ sudo nano /etc/hostname 

Edit hostname sudo nano /etc/hosts

Make sure your hostname maps to the ip like

127.0.1.1     hostname     alias 

If you have made changes in the previous steps then restart the server

$ sudo shutdown -r now

And again as reiterated by jay zipped the ambari log and share

@Jay Kumar SenSharma

grep 'jdbc' /etc/ambari-server/conf/ambari.properties custom.mysql.jdbc.name=mysql-connector-java-5.1.47.jar server.jdbc.connection-pool=c3p0 server.jdbc.connection-pool.acquisition-size=5 server.jdbc.connection-pool.idle-test-interval=7200 server.jdbc.connection-pool.max-age=0 server.jdbc.connection-pool.max-idle-time=14400 server.jdbc.connection-pool.max-idle-time-excess=0 server.jdbc.database=mysql server.jdbc.database_name=amabari_data server.jdbc.driver=com.mysql.jdbc.Driver server.jdbc.driver.path=/usr/lib/java/mysql-connector-java-5.1.47.jar server.jdbc.hostname=N9722 server.jdbc.port=3306 server.jdbc.rca.driver=com.mysql.jdbc.Driver server.jdbc.rca.url=jdbc:mysql://N9722:3306/amabari_data server.jdbc.rca.user.name=root server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:mysql://N9722:3306/amabari_data server.jdbc.user.name=root server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

AmbariLdapConfigurationProvider:126 - JPA initialized event received: JpaInitializedEvent{eventType=JPA_INITIALIZED} 2018-08-28 10:55:38,090 INFO [pool-4-thread-1] AmbariLdapConfigurationProvider:90 - Loading LDAP configuration ... 2018-08-28 10:55:38,090 INFO [pool-4-thread-1] AmbariLdapConfigurationProvider:92 - Actually loading LDAP configuration from DB ... 2018-08-28 10:55:38,187 INFO [main] AbstractPoolBackedDataSource:212 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 5, acquireRetryAttempts -> 30, acquireRetr yDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mch ange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> z8kfsx9xz8id6b1rajz85|6b695b06, debugUnreturnedConnectionStackTraces -> false, description -> null, driverC lass -> com.mysql.jdbc.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> z8kfsx9xz8id6b1rajz85|6b695b06, idleConnectionTestPeriod -> 7200, initialPoolSize -> 5, jdbcUrl -> jdbc:mysql://N9722:3306/amabari_data, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 14400, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 32, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, preferredTestQuery -> SELECT 1, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ] 2018-08-28 10:56:08,395 WARN [C3P0PooledConnectionPoolManager[identityToken->z8kfsx9xz8id6b1rajz85|6b695b06]-HelperThread-#2] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@26fccf15 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor50.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor49.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)

2018-08-28 10:56:08,395 WARN [C3P0PooledConnectionPoolManager[identityToken->z8kfsx9xz8id6b1rajz85|6b695b06]-HelperThread-#0] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@44b69099 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor50.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor49.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method)

i have change the hostname too... please let me know where am i doing it wrong

New Contributor

Check if /etc/hosts has private ips correctly entered in all hosts. It was causing issue in my case.