Created 01-29-2017 11:25 AM
Hi, I am getting below log message while trying to start ambari server in HDP 2.5.
[root@sandbox ~]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... No errors were found. ERROR: Exiting with exit code 1. REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information.
Then I search the root cause of issue in Google and found following solution.
Try deleting /var/lib/pgsql and restart ambari server. It worked for me
However i unable to delete the /var/lib/pgsql directory;
[root@sandbox ~]# sudo rmdir /var/lib/pgsql/* rmdir: failed to remove `/var/lib/pgsql/data': Directory not empty
Can anyone please help me to solve the above ambari-server issue and to delete pgsql directory.
Created 01-29-2017 12:53 PM
The error indicates database corruption in PostgreSQL.
I would suggest further investigating the corruption if you have data (other than Ambari's state) in the database.
If you are sure that you'd like to remove the contents of /var/lib/pgsql directory then you should use the following command:
sudo rm -rf /var/lib/pgsql/*
Created 01-29-2017 01:13 PM
Getting below error.
[root@sandbox /]# sudo rm -rf /var/lib/pgsql/* rm: cannot remove `/var/lib/pgsql/data/pg_stat_tmp': Invalid argument rm: cannot remove `/var/lib/pgsql/data/global': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_xlog': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_clog': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/1': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/11564': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/16384': Invalid argument rm: cannot remove `/var/lib/pgsql/data/base/11563': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_log': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_multixact/members': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_multixact/offsets': Invalid argument rm: cannot remove `/var/lib/pgsql/data/pg_subtrans': Invalid argument
Created 01-30-2017 07:28 PM
One can get invalid argument if a process is holding up the files.
You can try to stopping postgresql with 'service postgresql stop', then removing the files.
As @Jay SenSharma suggested 'ambari-server reset' could also solve this problem.
Created 01-29-2017 02:21 PM
Before attempting the postgres we need to make sure that is is stopped properly. It's PID does not exist anymore.
Also you might want to try resetting the ambari DB.
# ambari-server reset
- As you are planning to delete the postgres DB due to corruption. Hence it will be best to also try removing ambari completely and then install freshly as postgres has a dependency to postgres so it should install the postgres as well and while doing the "ambari-server setup" it will create a fresh "ambari" DB and will create necessary tables in the DB.
Created 03-21-2017 11:51 PM
The ambari-server reset is not necessarily the good solution. That command will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard. True. This is a sandbox, but in a production cluster ...