Ambari server startup error

[#1]

Hi,

I am getting following error message in the log file "/var/log/ambari-server/ambari-server.log" during ambari server startup.

I performed all the steps as mentioned in "https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/_using_ambari_with_postgresql.html".

14 Feb 2017 21:21:49,597 INFO [main] ClasspathScannerUtils:64 - Found class [class org.apache.ambari.server.audit.request.eventcreator.ViewPrivilegeEventCreator] 14 Feb 2017 21:21:51,229 ERROR [main] DBAccessorImpl:109 - Error while creating database accessor org.postgresql.util.PSQLException: FATAL: database "AMBARIDATABASE" does not exist at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:574) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:177) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:89)

[#2]

@Manish Anand

Can you share/attach latest ambari-server.log (or error snippet)

[#3]

@apappu

ambari-server-log.txt

PFA

[#4]

@Manish Anand

Just for testing try the following:

0. Stop ambari-server and take the Posgtres DB dump using "pg_dump" utility.

  ambari-server stop

1. Rename the database "AMBARIDATABASE" to lowercase "ambari" (or something else like "xyzdb")

  alter database "AMBARIDATABASE" RENAME TO "ambari";

2. Edit the "/etc/ambari-server/conf/ambari.properties" and change the database name to the new one.

  server.jdbc.database_name=ambari 

3. Start ambari-server.

  ambari-server start

.

[#5]

@Jay SenSharma

I performed the above steps. The server log is still showing the same error. I have in fact dropped the "AMBARIDATABASE". While performing the ambari server setup, I remember mentioning the "AMBARIDATABASE" and its schema and user details. Can you please let me know if this can be the issue and how to clean this up?

postgres=# drop database "ambaridatabase"; DROP DATABASE postgres=# \list List of databases Name | Owner | Encoding | Collation | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- ambari | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres : postgres=CTc/postgres : ambari=CTc/postgres

postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |

template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres : postgres=CTc/postgres

template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres : postgres=CTc/postgres

(4 rows)

[#6]

@Manish Anand

You mentioned that [The server log is still showing the same error. I have in fact dropped the "AMBARIDATABASE".]

Are you checking the latest log?

Can you clear the "/var/log/ambari-server/*" directory old logs and then freshly start ambari server sothat we can see the latest log.

Also please try the run the following command to findout from where is it getting that name ?

# cd /etc/ambari-server/conf/
# find . -name "*.*" | xargs grep -i "AMBARIDATABASE"

.

[#7]

@Jay SenSharma

Thanks for this step. I was able to locate the issue in "ambari.properties" file.

./ambari.properties:server.jdbc.rca.url=jdbc:postgresql://sandbox.hortonworks.com:5432/AMBARIDATABASE ./ambari.properties:server.jdbc.url=jdbc:postgresql://sandbox.hortonworks.com:5432/AMBARIDATABASE

I have modified the above entries and now pointing it to "ambari". The earlier error is resolved now but the server startup is failing in database consistency check. I have attached the log file for reference. Can you please take a look and let me know what might be missing?ambari-server-check-database.txt

[#8]

@Jay SenSharma

I tried to start the ambari server by skipping database consistency check (ambari-server start --skip-database-check) and got the following error.

15 Feb 2017 17:53:38,668 INFO [main] AmbariServer:913 - Getting the controller 15 Feb 2017 17:53:41,285 ERROR [main] AmbariServer:927 - Failed to run the Ambari Server Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: relation "metainfo" does not exist Position: 46 Error Code: 0 Call: SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?) bind => [1 parameter bound] Query: ReadObjectQuery(name="readMetainfoEntity" referenceClass=MetainfoEntity sql="SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?)") at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:684) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560) at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2055) at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)

How can I fix these issues? Can you please help? Is skipping database consistency check recommended?

[#9]

@Manish Anand

looks like your tables are not visible to ambari user. please login with that user and then try to see if you can do "select * from metainfo"

[#10]

Thanks @apappu

I have corrected the "ambari.properties" file with correct user and also updated the "password.dat" file.

Looks like the server started successfully but upon checking the server status the message says its not running.

[root@sandbox ~]# ambari-server start --skip-database-check

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources...

Ambari Server is starting with the database consistency check skipped. Do not make any changes to your cluster topology or perform a cluster upgrade until you correct the database consistency issues. See "/var/log/ambari-server/ambari-server-check-database.log" for more details on the consistency issues. Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start....................

Ambari Server 'start' completed successfully.

[root@sandbox ~]# ambari-server status

Using python /usr/bin/python Ambari-server status Ambari Server not running.

Stale PID File at: /var/run/ambari-server/ambari-server.pid

[#11]

I verified the log file and got the following error message:

15 Feb 2017 19:23:32,044 ERROR [main] AmbariJpaLocalTxnInterceptor:180 - [DETAILED ERROR] Rollback reason: Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: could not read block 0 of relation base/16384/16567: read only 0 of 8192 bytes Error Code: 0 Call: SELECT cluster_id, current_cluster_state, current_stack_id FROM clusterstate WHERE (cluster_id = ?) bind => [1 parameter bound] Query: ReadObjectQuery(name="clusterStateEntity" referenceClass=ClusterStateEntity )