Urgent support: Ambari-server can not start cause can not connect to mysql

[#1]

I can not start ambari server due to an errors:

2021-07-27 09:55:47,338  WARN [C3P0PooledConnectionPoolManager[identityToken->1br2om1ai1rsa6bt17v8a7t|338fc1d8]-HelperThread-#2] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@56b6d697 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
	at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
2021-07-27 09:55:47,340  WARN [C3P0PooledConnectionPoolManager[identityToken->1br2om1ai1rsa6bt17v8a7t|338fc1d8]-HelperThread-#2] BasicResourcePool:220 - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@69feb4d9 is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
2021-07-27 09:55:47,347 ERROR [main] AmbariServer:1119 - Failed to run the Ambari Server
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!
Error Code: 0
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:815)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:205)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:305)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:337)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:303)
	at com.google.inject.persist.jpa.JpaPersistService.begin(JpaPersistService.java:77)
	at com.google.inject.persist.jpa.AmbariJpaPersistService.begin(AmbariJpaPersistService.java:28)
	at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:40)
	at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkDBVersionCompatible(DatabaseConsistencyCheckHelper.java:230)
	at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1104)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!
Error Code: 0
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:207)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:760)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:265)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:731)
	... 9 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
	at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:146)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:144)
	... 14 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
	... 16 more
Caused by: java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
	at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

My repolist is as below: 

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: d36uatko69830t.cloudfront.net
 * extras: d36uatko69830t.cloudfront.net
 * updates: d36uatko69830t.cloudfront.net
repo id                                                                                            repo name                                                                                               status
Ambari-2.7.4.0-118                                                                                 Ambari Version - Ambari-2.7.4.0-118                                                                         13
HDP-3.1-GPL-repo-1                                                                                 HDP-3.1-GPL-repo-1                                                                                           4
HDP-3.1-repo-1                                                                                     HDP-3.1-repo-1                                                                                             201
HDP-3.1.4.0-315                                                                                    HDP Version - HDP-3.1.4.0-315                                                                              201
HDP-GPL-3.1.4.0-315                                                                                HDP-GPL Version - HDP-GPL-3.1.4.0-315                                                                        4
HDP-UTILS-1.1.0.22                                                                                 HDP-UTILS Version - HDP-UTILS-1.1.0.22                                                                      16
HDP-UTILS-1.1.0.22-repo-1                                                                          HDP-UTILS-1.1.0.22-repo-1                                                                                   16
base/7/x86_64                                                                                      CentOS-7 - Base                                                                                         10,072
extras/7/x86_64                                                                                    CentOS-7 - Extras                                                                                          498
mysql-connectors-community/x86_64                                                                  MySQL Connectors Community                                                                                 212
mysql-tools-community/x86_64                                                                       MySQL Tools Community                                                                                      132
mysql57-community/x86_64                                                                           MySQL 5.7 Community Server                                                                                 524
updates/7/x86_64                                                                                   CentOS-7 - Updates                                                                                       2,542
repolist: 14,435

Also: I install a mysql ambari database with user name below:

create database ambari;
create user ambari@'%' identified by 'Ctct@123';
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'mtnode.hdp.vn' WITH GRANT OPTION;
flush privileges;

But still the error existed,

Please help

[#2]

@Buithuy96 

First and foremost re-running these steps won't do damage to your cluster I assure you. What you have is purely a permissions issue

java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES)

Revalidate the MySQL connector

# yum install -y mysql-connector-java
# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Re-run the Ambari user setup

CREATE USER 'ambari'@'%' IDENTIFIED BY 'aCtct@123';
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'Ctct@123';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn' IDENTIFIED BY 'Ctct@123';
FLUSH PRIVILEGES;

Try restarting Ambari while tailing the ambari-server.log and share the contents, first reset the log before starting Ambari, to ensure you have a minimum log to delve through

# truncate --size 0 /var/logs/ambari-server/ambari-server.log

Restart your Ambari server and tail the logs

# tail -f /var/logs/ambari-server/ambari-server.log

Share the status