question Ambari server startup error in Support Questions

Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 03:55:59 GMT

Hi,

I am getting following error message in the log file "/var/log/ambari-server/ambari-server.log" during ambari server startup.

I performed all the steps as mentioned in "https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/_using_ambari_with_postgresql.html".

14 Feb 2017 21:21:49,597 INFO [main] ClasspathScannerUtils:64 - Found class [class org.apache.ambari.server.audit.request.eventcreator.ViewPrivilegeEventCreator] 14 Feb 2017 21:21:51,229 ERROR [main] DBAccessorImpl:109 - Error while creating database accessor org.postgresql.util.PSQLException: FATAL: database "AMBARIDATABASE" does not exist at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:574) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:177) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:89)

Re: Ambari server startup error

rrrajesh — Wed, 15 Feb 2017 06:34:31 GMT

It seems the postgres database with the name "AMBARIDATABASE" is not existing in your environment.

Crosscheck your ambari server config file. Usually the db name will be "ambari"

cat /etc/ambari-server/conf/ambari.properties | grep -i server.jdbc.database_name

server.jdbc.database_name=ambari

To verify the postgres database name available,

[root@conf]# su - postgres

-bash-4.1$ psql

psql (8.4.20)

Type "help" for help.

postgres=# \list

List of databases

-----------+----------+----------+-------------+-------------+-----------------------

: postgres=CTc/postgres

: ambari=CTc/postgres

: postgres=CTc/postgres

: postgres=CTc/postgres

(5 rows)

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 06:41:24 GMT

@Rajesh

I performed the check and I do see the database name "AMBARIDATABASE".

[root@sandbox ~]# cat /etc/ambari-server/conf/ambari.properties | grep -i server.jdbc.database_name server.jdbc.database_name=AMBARIDATABASE

[root@sandbox ~]# su - postgres

-bash-4.1$ psql

psql (8.4.20)

Type "help" for help.

postgres=# \list

(5 rows)

postgres=#

Re: Ambari server startup error

rrrajesh — Wed, 15 Feb 2017 06:47:51 GMT

@Manish

The database name is case sensitive.

Can you change it to "ambaridatabase" in your /etc/ambari-server/conf/ambari.properties file and restart ambari server.

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 08:31:22 GMT

Hi,

I performed the modification as you mentioned but its still throwing the same error message.

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 08:35:29 GMT

The interesting part - Even after changing the parameter to 'ambari' in the properties file, the error log is still throwing the error on 'AMBARIDATABASE'.

Re: Ambari server startup error

apappu — Wed, 15 Feb 2017 08:39:10 GMT

Manish Anand

what is the Ambari version? if it is 2.4.0 then kill Ambari server process and start again.

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 08:43:40 GMT

The version is 2.4.0

[root@sandbox ~]# rpm -qa|grep ambari-server ambari-server-2.4.0.0-1225.x86_64

The ambari server isn't starting up. Which process do you you want me to kill??

Re: Ambari server startup error

apappu — Wed, 15 Feb 2017 08:47:29 GMT

Manish Anand

There is a bug in 2.4.0 - server some times does not get restart when you do "ambari-server stop"

Run

ps -aef | grep ambari-server

it gives you PID - kill -9 PID

then start again and check

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 10:56:21 GMT

There isn't any process running. What shall I do next?

[root@sandbox ~]# ps -aef | grep ambari-server

root 17872 17671 0 02:54 pts/0 00:00:00 grep ambari-server

[root@sandbox ~]#

Re: Ambari server startup error

apappu — Wed, 15 Feb 2017 10:57:54 GMT

@Manish Anand

Can you share/attach latest ambari-server.log (or error snippet)

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 11:08:40 GMT

@apappu

ambari-server-log.txt

PFA

Re: Ambari server startup error

jsensharma — Wed, 15 Feb 2017 11:21:50 GMT

@Manish Anand

Just for testing try the following:

0. Stop ambari-server and take the Posgtres DB dump using "pg_dump" utility.

  ambari-server stop

1. Rename the database "AMBARIDATABASE" to lowercase "ambari" (or something else like "xyzdb")

  alter database "AMBARIDATABASE" RENAME TO "ambari";

2. Edit the "/etc/ambari-server/conf/ambari.properties" and change the database name to the new one.

  server.jdbc.database_name=ambari

3. Start ambari-server.

  ambari-server start

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 12:06:49 GMT

@Jay SenSharma

I performed the above steps. The server log is still showing the same error. I have in fact dropped the "AMBARIDATABASE". While performing the ambari server setup, I remember mentioning the "AMBARIDATABASE" and its schema and user details. Can you please let me know if this can be the issue and how to clean this up?

(4 rows)

Re: Ambari server startup error

jsensharma — Wed, 15 Feb 2017 12:31:39 GMT

@Manish Anand

You mentioned that [The server log is still showing the same error. I have in fact dropped the "AMBARIDATABASE".]

Are you checking the latest log?

Can you clear the "/var/log/ambari-server/*" directory old logs and then freshly start ambari server sothat we can see the latest log.

Also please try the run the following command to findout from where is it getting that name ?

# cd /etc/ambari-server/conf/
# find . -name "*.*" | xargs grep -i "AMBARIDATABASE"

Re: Ambari server startup error

manish_raj_anan — Wed, 15 Feb 2017 23:55:27 GMT

@Jay SenSharma

Thanks for this step. I was able to locate the issue in "ambari.properties" file.

./ambari.properties:server.jdbc.rca.url=jdbc:postgresql://sandbox.hortonworks.com:5432/AMBARIDATABASE ./ambari.properties:server.jdbc.url=jdbc:postgresql://sandbox.hortonworks.com:5432/AMBARIDATABASE

I have modified the above entries and now pointing it to "ambari". The earlier error is resolved now but the server startup is failing in database consistency check. I have attached the log file for reference. Can you please take a look and let me know what might be missing?ambari-server-check-database.txt

Re: Ambari server startup error

manish_raj_anan — Thu, 16 Feb 2017 02:02:18 GMT

@Jay SenSharma

I tried to start the ambari server by skipping database consistency check (ambari-server start --skip-database-check) and got the following error.

15 Feb 2017 17:53:38,668 INFO [main] AmbariServer:913 - Getting the controller 15 Feb 2017 17:53:41,285 ERROR [main] AmbariServer:927 - Failed to run the Ambari Server Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: relation "metainfo" does not exist Position: 46 Error Code: 0 Call: SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?) bind => [1 parameter bound] Query: ReadObjectQuery(name="readMetainfoEntity" referenceClass=MetainfoEntity sql="SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?)") at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:684) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560) at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2055) at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)

How can I fix these issues? Can you please help? Is skipping database consistency check recommended?

Re: Ambari server startup error

apappu — Thu, 16 Feb 2017 02:08:41 GMT

@Manish Anand

looks like your tables are not visible to ambari user. please login with that user and then try to see if you can do "select * from metainfo"

Re: Ambari server startup error

manish_raj_anan — Thu, 16 Feb 2017 03:22:15 GMT

Thanks @apappu

I have corrected the "ambari.properties" file with correct user and also updated the "password.dat" file.

Looks like the server started successfully but upon checking the server status the message says its not running.

[root@sandbox ~]# ambari-server start --skip-database-check

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources...

Ambari Server is starting with the database consistency check skipped. Do not make any changes to your cluster topology or perform a cluster upgrade until you correct the database consistency issues. See "/var/log/ambari-server/ambari-server-check-database.log" for more details on the consistency issues. Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start....................

Ambari Server 'start' completed successfully.

[root@sandbox ~]# ambari-server status

Using python /usr/bin/python Ambari-server status Ambari Server not running.

Stale PID File at: /var/run/ambari-server/ambari-server.pid

Re: Ambari server startup error

manish_raj_anan — Thu, 16 Feb 2017 03:25:37 GMT

I verified the log file and got the following error message:

15 Feb 2017 19:23:32,044 ERROR [main] AmbariJpaLocalTxnInterceptor:180 - [DETAILED ERROR] Rollback reason: Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: could not read block 0 of relation base/16384/16567: read only 0 of 8192 bytes Error Code: 0 Call: SELECT cluster_id, current_cluster_state, current_stack_id FROM clusterstate WHERE (cluster_id = ?) bind => [1 parameter bound] Query: ReadObjectQuery(name="clusterStateEntity" referenceClass=ClusterStateEntity )