Urgent support: Ambari-server can not start cause can not connect to mysql

Buithuy96 — Tue, 27 Jul 2021 03:11:00 GMT

I can not start ambari server due to an errors:

2021-07-27 09:55:47,338 WARN [C3P0PooledConnectionPoolManager[identityToken->1br2om1ai1rsa6bt17v8a7t|338fc1d8]-HelperThread-#2] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@56b6d697 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) 2021-07-27 09:55:47,340 WARN [C3P0PooledConnectionPoolManager[identityToken->1br2om1ai1rsa6bt17v8a7t|338fc1d8]-HelperThread-#2] BasicResourcePool:220 - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@69feb4d9 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2021-07-27 09:55:47,347 ERROR [main] AmbariServer:1119 - Failed to run the Ambari Server javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database! Error Code: 0 at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:815) at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:205) at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:305) at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:337) at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:303) at com.google.inject.persist.jpa.JpaPersistService.begin(JpaPersistService.java:77) at com.google.inject.persist.jpa.AmbariJpaPersistService.begin(AmbariJpaPersistService.java:28) at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:40) at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkDBVersionCompatible(DatabaseConsistencyCheckHelper.java:230) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1104) Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database! Error Code: 0 at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316) at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147) at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162) at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:207) at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:760) at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:265) at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:731) ... 9 more Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database! at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:146) at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:144) ... 14 more Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source. at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685) ... 16 more Caused by: java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

My repolist is as below:

Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: d36uatko69830t.cloudfront.net * extras: d36uatko69830t.cloudfront.net * updates: d36uatko69830t.cloudfront.net repo id repo name status Ambari-2.7.4.0-118 Ambari Version - Ambari-2.7.4.0-118 13 HDP-3.1-GPL-repo-1 HDP-3.1-GPL-repo-1 4 HDP-3.1-repo-1 HDP-3.1-repo-1 201 HDP-3.1.4.0-315 HDP Version - HDP-3.1.4.0-315 201 HDP-GPL-3.1.4.0-315 HDP-GPL Version - HDP-GPL-3.1.4.0-315 4 HDP-UTILS-1.1.0.22 HDP-UTILS Version - HDP-UTILS-1.1.0.22 16 HDP-UTILS-1.1.0.22-repo-1 HDP-UTILS-1.1.0.22-repo-1 16 base/7/x86_64 CentOS-7 - Base 10,072 extras/7/x86_64 CentOS-7 - Extras 498 mysql-connectors-community/x86_64 MySQL Connectors Community 212 mysql-tools-community/x86_64 MySQL Tools Community 132 mysql57-community/x86_64 MySQL 5.7 Community Server 524 updates/7/x86_64 CentOS-7 - Updates 2,542 repolist: 14,435

Also: I install a mysql ambari database with user name below:

create database ambari; create user ambari@'%' identified by 'Ctct@123'; GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' WITH GRANT OPTION; GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'mtnode.hdp.vn' WITH GRANT OPTION; flush privileges;

But still the error existed,

Please help

Re: Urgent support: Ambari-server can not start cause can not connect to mysql

Shelton — Fri, 30 Jul 2021 21:48:39 GMT

@Buithuy96

First and foremost re-running these steps won't do damage to your cluster I assure you. What you have is purely a permissions issue

java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES)

Revalidate the MySQL connector

# yum install -y mysql-connector-java # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Re-run the Ambari user setup

CREATE USER 'ambari'@'%' IDENTIFIED BY 'aCtct@123'; CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'Ctct@123'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn' IDENTIFIED BY 'Ctct@123'; FLUSH PRIVILEGES;

Try restarting Ambari while tailing the ambari-server.log and share the contents, first reset the log before starting Ambari, to ensure you have a minimum log to delve through

# truncate --size 0 /var/logs/ambari-server/ambari-server.log

Restart your Ambari server and tail the logs

# tail -f /var/logs/ambari-server/ambari-server.log

Share the status

question Urgent support: Ambari-server can not start cause can not connect to mysql in Support Questions

Urgent support: Ambari-server can not start cause can not connect to mysql

Re: Urgent support: Ambari-server can not start cause can not connect to mysql