Unable to start Ambari server in HDP 2.5

[#1]

Hi, I am getting below log message while trying to start ambari server in HDP 2.5.

[root@sandbox ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
No errors were found.
ERROR: Exiting with exit code 1.
REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information.

ambari-server-check-database.txt

ambari-server.txt

Then I search the root cause of issue in Google and found following solution.

Try deleting /var/lib/pgsql and restart ambari server. It worked for me

However i unable to delete the /var/lib/pgsql directory;

[root@sandbox ~]# sudo rmdir /var/lib/pgsql/* 

rmdir: failed to remove `/var/lib/pgsql/data': Directory not empty

Can anyone please help me to solve the above ambari-server issue and to delete pgsql directory.

Thanks,

[#2]

The error indicates database corruption in PostgreSQL.

I would suggest further investigating the corruption if you have data (other than Ambari's state) in the database.

If you are sure that you'd like to remove the contents of /var/lib/pgsql directory then you should use the following command:

sudo rm -rf /var/lib/pgsql/*

[#3]

Getting below error.

[root@sandbox /]# sudo rm -rf /var/lib/pgsql/*
rm: cannot remove `/var/lib/pgsql/data/pg_stat_tmp': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/global': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_xlog': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_clog': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/base/1': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/base/11564': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/base/16384': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/base/11563': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_log': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_multixact/members': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_multixact/offsets': Invalid argument
rm: cannot remove `/var/lib/pgsql/data/pg_subtrans': Invalid argument

[#4]

One can get invalid argument if a process is holding up the files.

You can try to stopping postgresql with 'service postgresql stop', then removing the files.

As @Jay SenSharma suggested 'ambari-server reset' could also solve this problem.

[#5]

@Prasanta Sahoo

Before attempting the postgres we need to make sure that is is stopped properly. It's PID does not exist anymore.

Also you might want to try resetting the ambari DB.

# ambari-server reset

- As you are planning to delete the postgres DB due to corruption. Hence it will be best to also try removing ambari completely and then install freshly as postgres has a dependency to postgres so it should install the postgres as well and while doing the "ambari-server setup" it will create a fresh "ambari" DB and will create necessary tables in the DB.

https://community.hortonworks.com/questions/1110/how-to-completely-remove-uninstall-ambari-and-hdp.h...

.

[#6]

The ambari-server reset is not necessarily the good solution. That command will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard. True. This is a sandbox, but in a production cluster ...