Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ambari server problem

ambari server problem

Explorer

Hi

 

when i start my ambari server 

 

ambari-server restart


Using python /usr/bin/python
Restarting ambari-server
Ambari Server is not running
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..........................................ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.

 

ambari-server status

 

Using python /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid

 

 

Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!
Error Code: 0
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:316)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:147)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:207)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:760)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:265)
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:731)
... 9 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:146)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:144)
... 14 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1469)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 16 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor41.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)

15 REPLIES 15

Re: ambari server problem

Super Mentor

@Manoj690 

As we see the following error

Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Which basically indicates that either your MySQL is not running fully or has some issues while connecting. 

So please check if MySQL server is running fine?  

Verify if you are using correct MySQL Host & Port.

Try running the following kind of simple connectivity test:

# /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar  org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://dbhost.example.com:3306/ambari" "ambari" "bigdata" com.mysql.jdbc.Driver


You can get the JDBC url/username & password details by looking at the following properties file:

# grep 'jdbc' /etc/ambari-server/conf/ambari.properties

 

Re: ambari server problem

Explorer

that connectivity check command still running , how much time it will take 

 

grep 'jdbc' /etc/ambari-server/conf/ambari.properties


server.jdbc.connection-pool=c3p0
server.jdbc.connection-pool.acquisition-size=5
server.jdbc.connection-pool.idle-test-interval=7200
server.jdbc.connection-pool.max-age=0
server.jdbc.connection-pool.max-idle-time=14400
server.jdbc.connection-pool.max-idle-time-excess=0
server.jdbc.database=mysql
server.jdbc.database_name=hive
server.jdbc.driver=com.mysql.jdbc.Driver
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
server.jdbc.hostname=gaian-lap386.com
server.jdbc.port=8080
server.jdbc.rca.driver=com.mysql.jdbc.Driver
server.jdbc.rca.url=jdbc:mysql://gaian-lap386.com:8080/hive
server.jdbc.rca.user.name=hive
server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat
server.jdbc.url=jdbc:mysql://gaian-lap386.com:8080/hive
server.jdbc.user.name=hive
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

Re: ambari server problem

Super Mentor

@Manoj690 

 

It looks bit strange that your ambari DB name is "hive

MySQL DB port is 8080  (usually it is kept as 3306 ... ofcourse you can choose whatever port you want to run MySQL but i am just curious to know if you are using the correct DB to connect)

server.jdbc.database=mysql
server.jdbc.database_name=hive
server.jdbc.url=jdbc:mysql://gaian-lap386.com:8080/hive
server.jdbc.user.name=hive
server.jdbc.hostname=gaian-lap386.com
server.jdbc.port=8080

 
If you are very sure that above DB setting you want to use for your Ambari DB then please check if that is accessible from Ambari Server host or not?

# /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar  org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://gaian-lap386.com:8080/hive" "hive" "hive" com.mysql.jdbc.Driver


Are you able to run the above kind of command from ambari server host to connect to MySQL ?
Please enter the correct JDK and DB password in the above command to test the mysql connectivity.

Re: ambari server problem

Explorer

in MYSQL i created user name and database name is hive only. 

i run above command it shows

ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Re: ambari server problem

Super Mentor

@Manoj690 

As mentioned earlier that the following error looks more specific to MySQL

ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

 

So next troubleshooting is needed from MySQL side.

Please verify if your MySQL DB is up and running and listening to port 8080
Try restarting MySQL

# service mysqld status
# netstat -tnlpa | grep 8080
# service mysqld restart

 

Form Ambari Server host please check the MySQL Port access 

# telnet gaian-lap386.com 8080
(OR)
# nc -v gaian-lap386.com 8080


Please check and share the MySQL log.  Usually it is at "/var/log/mysql/mysql.log"

From Ambari Server host, Are you able to connect to MySQL using native client utility "mysql" and able to list the tables ?  

# mysql -u $AMBARI_DB_USER_NAME $AMBARI_DBNAME -p$AMBARI_DB_PASSWORD_MYSQL --execute="SELECT * FROM hosts"

Example:
# mysql -u hive hive -p$AMBARI_DB_PASSWORD_MYSQL --host gaian-lap386.com --port 8080 --execute="SELECT * FROM hosts"



Re: ambari server problem

Explorer

service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-08-20 17:20:17 IST; 51min ago
Process: 982 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=0/SUCCESS)
Process: 915 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 984 (mysqld)
Tasks: 28 (limit: 4915)
CGroup: /system.slice/mysql.service
└─984 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

Aug 20 17:20:06 gaian-lap386.com systemd[1]: Starting MySQL Community Server...
Aug 20 17:20:17 gaian-lap386.com systemd[1]: Started MySQL Community Server.

 


netstat -tnlpa | grep 8080
service mysql restart

 

 telnet gaian-lap386.com 8080
Trying 192.168.24.32...
telnet: Unable to connect to remote host: Connection refused

mysql -u hive hive -p $gaian_MYSQL --host gaian-lap386.com --port 8080 --execute="SELECT * FROM hosts"
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on 'gaian-lap386.com' (111)

Re: ambari server problem

Explorer

the port has changed,, how to change this port

server.jdbc.port=808
server.jdbc.rca.driver=com.mysql.jdbc.Driver
server.jdbc.rca.url=jdbc:mysql://gaian-lap386.com:808/hive
server.jdbc.rca.user.name=hive

Highlighted

Re: ambari server problem

Mentor

@Manoj690 

 

Please follow the below steps remember to skip steps you've already executed on the ambari database host.

 

# Install the mysql connector [optional if already executed]

yum install -y mysql-connector-java
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

 

# Enable auto start on boot [optional if already executed]

sudo systemctl start mysqld [if not started]
sudo systemctl enable mysqld


I think you didn't create the Ambari backend database, please follow the steps below as root

Assumption
root password=w3lc0m31
ambari username = ambari
ambari user password=ambari

 

mysql -u root -pw3lc0m31
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'gaian-lap386.com';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'gaian-lap386.com' IDENTIFIED BY 'ambari';
FLUSH PRIVILEGES;

# Log on as Ambari user and create the Ambari DB

mysql -u ambari -pambari
CREATE DATABASE ambari;
USE ambari;
SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
exit;


After successfully running the below switch to the  ambari host CLI, the default mysql database port is 3306 anything else is not acceptable. The below command should pull parameters macting the below


# grep 'jdbc' /etc/ambari-server/conf/ambari.properties

server.jdbc.database=mysql
server.jdbc.database_name=ambari
server.jdbc.driver=com.mysql.jdbc.Driver
server.jdbc.hostname=gaian-lap386.com
server.jdbc.port=3306
server.jdbc.rca.driver=com.mysql.jdbc.Driver
server.jdbc.rca.url=jdbc:mysql://gaian-lap386.com:3306/ambari
server.jdbc.rca.user.name=ambari
server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat
server.jdbc.url=jdbc:mysql://gaian-lap386.com:3306/ambari
server.jdbc.user.name=ambari


# Start Ambari as root user

# ambari-server start

This time it should succeed.



# To configure hive, oozie, ranger and rangerkms use this template


mysql -uroot -pw3lc0m31

create database hive;
grant all privileges on hive.* to 'hive'@'localhost' identified by 'hive';
grant all privileges on hive.* to 'hive'@'%gaian-lap386.com' identified by 'hive';

 

Substitute the values and names for the subsequent components after the Ambari startup ensure these values are matched in the subsequent component database setup


Hope that helps

Re: ambari server problem

Explorer

grep 'jdbc' /etc/ambari-server/conf/ambari.properties
server.jdbc.connection-pool=c3p0
server.jdbc.connection-pool.acquisition-size=5
server.jdbc.connection-pool.idle-test-interval=7200
server.jdbc.connection-pool.max-age=0
server.jdbc.connection-pool.max-idle-time=14400
server.jdbc.connection-pool.max-idle-time-excess=0
server.jdbc.database=mysql
server.jdbc.database_name=ambari
server.jdbc.driver=com.mysql.jdbc.Driver
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
server.jdbc.hostname=gaian-lap386.com
server.jdbc.port=3306
server.jdbc.rca.driver=com.mysql.jdbc.Driver
server.jdbc.rca.url=jdbc:mysql://gaian-lap386.com:3306/ambari
server.jdbc.rca.user.name=ambari
server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat
server.jdbc.url=jdbc:mysql://gaian-lap386.com:3306/ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

 

 

i got this but still it shows same error

 

ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..........................................ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.