Ambari doesnt start after postgresql Upgrade

[#1]

We have a four-node Hadoop cluster with HDP 2.4 and Kerberos installed in it. As this is our production cluster, We wanted to have HA for all the services including the PostgreSQL database which is used by Hive, Ambari, and Oozie for storing the metadata. However, the version of our PostgreSQL, which is 8.4.2 doesn't support the inbuilt feature(stream replication) of Postgres.

So, we have decided to upgrade PostgreSQL to a version(9.3) ambari supports.

I followed this link to upgrade the Postgres. Everything went well. Expect that, we are getting the following error when restarting ambari server.

Ambari Server running with administrator privileges.
Running initdb: This may take upto a minute.
Data directory is not empty!
[FAILED]

Could someone help?

Thanks.

[#2 · Accepted solution]

@vinay kumar

You have upgraded the PostgresSQL to 9.3, But do you see that the "ambari" DB still exist there?

Please check if you see the Ambari DB Tables there?

# psql -U ambari ambari
Password for user ambari:  bigdata
psql (9.5.5)
Type "help" for help.
ambari=> \dt+

If the Postgres SQL 9.3 has the ambari DB (upgraded from Postgres SQL 8.1) then you shoudl see all the existing tables and data.

Now If the postgressql hostname or port is changed during upgrade then you shoudl run the "smbari-server setup" again and this time you should choose the "" option and say "y"

Enter advanced database configuration [y/n] (n)? y

And then choose the hostname & port properly and do nto use the "[1] - PostgreSQL (Embedded)" option while setup else ambari will try to reinitialize the DB and as the Ambari DB already exist so it might cause issues.

You should choose Option

[4] - PostgreSQL

Example: (Before doing the ambari-server setup again Please take Ambari DB dump and keep safely)

# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (4): 4
Hostname (standaloneambari.example.com): standaloneambari.example.com 
Port (5432): 
Database name (ambari): 
Postgres schema (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Configuring ambari database...
Configuring remote database connection properties...
WARNING:
 Before starting Ambari Server, you must run the following DDL against 
the database to create the schema: 
/var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)

.

[#3]

yes, I have restored the tables using the backup I have taken before I stopped old PostgreSQL(8.4). reinitializing it with option 4 solved the issue. Thank you.