Ambari-server start error

[#1]

I am trying to install HDP 3.1 with Ambari 2.7 following this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/set_up_the_a...

All software installation completed and mysql database had been created. I then setup ambari with "ambari setup" followed by "ambari start". But I got follwing errors. I had listed the screen output for ambari setup and part of ambari-server.log to show the error message. Please help.

Thanks,

Harry

Setup and start ambari-server

[root@msl-dpe-perf80-100g resources]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? 
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...

Do you want to change Oracle JDK [y/n] (n)? Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (localhost): msl-dpe-perf80-100g.msl.lab
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? 
Extracting system views...
.....
Ambari repo file contains latest json url <a href="<a href="http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json" target="_blank">http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json</a>" target="_blank"><a href="http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json</a" target="_blank">http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json</a</a>>, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@msl-dpe-perf80-100g resources]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start........................................ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process has stopped. Please check the logs for more information.
[root@msl-dpe-perf80-100g resources]# view /var/log/ambari-server/ambari-server.log

ambari-server.log errors:

2019-08-21 15:30:03,832  INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.UpgradeItemEventCreator
2019-08-21 15:30:03,832  INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.UserEventCreator
2019-08-21 15:30:03,832  INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.ValidationIgnoreEventCreator
2019-08-21 15:30:03,832  INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.ViewInstanceEventCreator
2019-08-21 15:30:03,832  INFO [main] AuditLoggerModule:82 - Binding audit event creator class org.apache.ambari.server.audit.request.eventcreator.ViewPrivilegeEventCreator
2019-08-21 15:30:04,856  INFO [main] HostRoleCommandDAO:277 - Host role command status summary cache enabled !
2019-08-21 15:30:04,858  INFO [main] TransactionalLock$LockArea:121 - LockArea HRC_STATUS_CACHE is enabled
2019-08-21 15:30:05,081  INFO [main] AmbariServerConfigurationProvider:68 - Registered org.apache.ambari.server.ldap.service.AmbariLdapConfigurationProvider in event publisher
2019-08-21 15:30:05,584  INFO [main] CredentialStoreServiceImpl:60 - Initialized the temporary credential store. KeyStore entries will be retained for 90 minutes and will be actively purged
2019-08-21 15:30:05,593  INFO [main] LockFactory:54 - Lock profiling is disabled
2019-08-21 15:30:05,594  INFO [main] AmbariServer:1083 - Getting the controller
2019-08-21 15:30:06,282  INFO [pool-5-thread-1] AmbariServerConfigurationProvider:98 - JPA initialized event received: JpaInitializedEvent{eventType=JPA_INITIALIZED}
2019-08-21 15:30:06,282  INFO [pool-5-thread-1] AmbariServerConfigurationProvider:122 - Loading ldap-configuration configuration data
2019-08-21 15:30:06,377  INFO [main] AbstractPoolBackedDataSource:212 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 5, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> 2vsgnka41ekyetann3ejj|57bc27f5, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> 2vsgnka41ekyetann3ejj|57bc27f5, idleConnectionTestPeriod -> 7200, initialPoolSize -> 5, jdbcUrl -> jdbc:mysql://msl-dpe-perf80-100g.msl.lab:3306/ambari, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 14400, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 32, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, preferredTestQuery -> SELECT 1, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ]
2019-08-21 15:30:36,719  WARN [C3P0PooledConnectionPoolManager[identityToken->2vsgnka41ekyetann3ejj|57bc27f5]-HelperThread-#1] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@30120b4 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
        at sun.reflect.GeneratedConstructorAccessor48.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.Util.getInstance(Util.java:386)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2570)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:839)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
        at sun.reflect.GeneratedConstructorAccessor42.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350)

[#2]

@harry_li 

Looks like due to some issue Ambari is not able to connect to MySQL and hence the Connection Pool is not getting populated.

Most probably it might be some issue with MySQL / Network issue (firewall..etc)/ Misconfiguration like incorrect host/port in ambari configs for MySQL host/port ...etc

So can you please test the connection by running this simple command from ambari server host to see if you get a Successful connection creation response?

# /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://xxxxxxxxxxx:3306/ambari" "ambari" "bigdata" com.mysql.jdbc.Driver

Please use your own MySQL username and passwords in the above query to test the connection. I am using default password as "bigdata" and DB user name as "ambari" in the above example query.

If it fails to connect to mysql then Pleas echeck if at n/w level the Port access is allowed to MySQL Server from Ambari Server host?

# telnet $MYSQL_FQDN 3306
(OR)
# nc -v $MYSQL_FQDN 3306

If possible then can you please try restarting your MysQL server once and then retry.

[#3]

Thanks for the quick response. I had firewall disabled and nc -v command seems working. 

[root@msl-dpe-perf80-100g resources]# java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://msl-dpe-perf80-100g.msl.lab:3306/ambari" "ambari" "bigdata" com.mysql.jdbc.Driver
ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.

[root@msl-dpe-perf80-100g resources]# firewall-cmd --state
not running

[root@msl-dpe-perf80-100g resources]# telnet msl-dpe-perf80-100g.msl.lab 3306
Trying 10.100.10.80...
Connected to msl-dpe-perf80-100g.msl.lab.
Escape character is '^]'.
J
8.0.17\ufffdv2RS3\ufffdKwrx?^]7qj6caching_sha2_passwordConnection closed by foreign host.

[root@msl-dpe-perf80-100g resources]# nc -v  msl-dpe-perf80-100g.msl.lab 3306
Ncat: Version 6.40 ( <a href="http://nmap.org/ncat" target="_blank">http://nmap.org/ncat</a> )
Ncat: Connected to 10.100.10.80:3306.
J
8.0.17\ufffdzZbd\ufffd\ufffd\ufffd\ufffd\ufffds_N}6+X?[|^caching_sha2_password

Ncat: Broken pipe.

From command line, I can access the mysql. See below for detail

[root@msl-dpe-perf80-100g resources]# mysql -uambari -pbigdata
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 172
Server version: 8.0.17 MySQL Community Server - GPL

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| ambari             |
| druid              |
| information_schema |
| mysql              |
| performance_schema |
| registry           |
| streamline         |
| superset           |
| sys                |
+--------------------+
9 rows in set (0.01 sec)

mysql> use ambari;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-------------------------------+
| Tables_in_ambari              |
+-------------------------------+
| ClusterHostMapping            |
| QRTZ_BLOB_TRIGGERS            |
| QRTZ_CALENDARS                |
| QRTZ_CRON_TRIGGERS            |
| QRTZ_FIRED_TRIGGERS           |
| QRTZ_JOB_DETAILS              |
| QRTZ_LOCKS                    |
| QRTZ_PAUSED_TRIGGER_GRPS      |
| QRTZ_SCHEDULER_STATE          |
| QRTZ_SIMPLE_TRIGGERS          |
| QRTZ_SIMPROP_TRIGGERS         |
| QRTZ_TRIGGERS                 |
| adminpermission               |
| adminprincipal                |
| adminprincipaltype            |
| adminprivilege                |
| adminresource                 |
| adminresourcetype             |
| alert_current                 |
| alert_definition              |
| alert_group                   |
| alert_group_target            |
| alert_grouping                |
| alert_history                 |
| alert_notice                  |
| alert_target                  |
| alert_target_states           |
| ambari_configuration          |
| ambari_operation_history      |
| ambari_sequences              |
| artifact                      |
| blueprint                     |
| blueprint_configuration       |
| blueprint_setting             |
| clusterconfig                 |
| clusters                      |
| clusterservices               |
| clusterstate                  |
| confgroupclusterconfigmapping |
| configgroup                   |
| configgrouphostmapping        |
| execution_command             |
| extension                     |
| extensionlink                 |
| host_role_command             |
| host_version                  |
| hostcomponentdesiredstate     |
| hostcomponentstate            |
| hostconfigmapping             |
| hostgroup                     |
| hostgroup_component           |
| hostgroup_configuration       |
| hosts                         |
| hoststate                     |
| kerberos_descriptor           |
| kerberos_keytab               |
| kerberos_keytab_principal     |
| kerberos_principal            |
| key_value_store               |
| kkp_mapping_service           |
| metainfo                      |
| permission_roleauthorization  |
| remoteambaricluster           |
| remoteambariclusterservice    |
| repo_applicable_services      |
| repo_definition               |
| repo_os                       |
| repo_tags                     |
| repo_version                  |
| request                       |
| requestoperationlevel         |
| requestresourcefilter         |
| requestschedule               |
| requestschedulebatchrequest   |
| role_success_criteria         |
| roleauthorization             |
| servicecomponent_version      |
| servicecomponentdesiredstate  |
| serviceconfig                 |
| serviceconfighosts            |
| serviceconfigmapping          |
| servicedesiredstate           |
| setting                       |
| stack                         |
| stage                         |
| topology_host_info            |
| topology_host_request         |
| topology_host_task            |
| topology_hostgroup            |
| topology_logical_request      |
| topology_logical_task         |
| topology_request              |
| upgrade                       |
| upgrade_group                 |
| upgrade_history               |
| upgrade_item                  |
| user_authentication           |
| users                         |
| viewentity                    |
| viewinstance                  |
| viewinstancedata              |
| viewinstanceproperty          |
| viewmain                      |
| viewparameter                 |
| viewresource                  |
| viewurl                       |
| widget                        |
| widget_layout                 |
| widget_layout_user_widget     |
+-------------------------------+
109 rows in set (0.00 sec)

mysql>  SELECT User FROM mysql.user;
+------------------+
| User             |
+------------------+
| ambari           |
| druid            |
| rangerdba        |
| registry         |
| streamline       |
| superset         |
| ambari           |
| mysql.infoschema |
| mysql.session    |
| mysql.sys        |
| rangerdba        |
| root             |
| ambari           |
| rangerdba        |
+------------------+
14 rows in set (0.00 sec)

mysql> SHOW GRANTS FOR 'ambari'@'msl-dpe-perf80-100g.msl.lab';
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for ambari@msl-dpe-perf80-100g.msl.lab                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `ambari`@`msl-dpe-perf80-100g.msl.lab`                                                                                       |
| GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,GROUP_REPLICATION_ADMIN,INNODB_REDO_LOG_ARCHIVE,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `ambari`@`msl-dpe-perf80-100g.msl.lab` |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.01 sec)

[#4 · Accepted solution]

@harry_li 

So is your Ambari Server running fine now?  Please share the ambari server log if you see the same MySQL error.

Also for tested and certified version of MySQL that can be used with Ambari please refer to the following link :  https://supportmatrix.hortonworks.com/

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#5]

Found the problem. It is mysql version mismatched.