Support Questions
Find answers, ask questions, and share your expertise

ambari server start fails after a restore to another vm Centos 6.8. postgres db Please review procedure and let me know what i am missing

Explorer

On the ambari server . Centos 6.8

ambari-server backup
Using python /usr/bin/python
Backing up Ambari File System state... *this will not backup the server database*
Backup requested.
No path specified. Will use /var/lib/ambari-server/Ambari_State_Backup.zip
Backup process initiated.
Creating zip file...
Zip file created at /var/lib/ambari-server/Ambari_State_Backup.zip
Backup complete.
Ambari Server 'backup' completed successfully.

[root@bob234hwxmgmt0 ~]# cp /var/lib/ambari-server/Ambari_State_Backup.zip /bkup

[root@bob234hwxmgmt0 ~]# ambari-server stop
Using python /usr/bin/python
Stopping ambari-server
Ambari Server stopped

[root@bob234hwxmgmt0 ~]# /usr/bin/pg_dump -U ambari ambari > /bkup/ambari.sql
Password:
[root@bob234hwxmgmt0 ~]# ls -l /bkup
total 1395896
-rw-r--r-- 1 root root 11794132 Oct 27 14:38 ambari.sql
-rw-r--r-- 1 root root 1417591774 Oct 27 14:36 Ambari_State_Backup.zip

On the new server I restore the ambari db and the ambari server tails to start
ambari-server stop
Using python /usr/bin/python
Stopping ambari-server
Ambari Server stopped

[root@bob234hwxworker5 ~]# su - postgres
[postgres@bob234hwxworker5 ~]$ psql
psql (8.4.20)
Type "help" for help.

postgres=# psql
postgres-# \l
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
ambari | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres
: postgres=CTc/postgres
: ambari=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
(4 rows)


postgres=# drop database ambari;
DROP DATABASE
postgres=# \l
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
(3 rows)

postgres=# create database ambari;
CREATE DATABASE

postgres=# psql -d ambari -f /bkup/hwx/ambari.sql
postgres-# \l
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
ambari | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
(4 rows)

postgres-# \q
[postgres@bob234hwxworker5 ~]$ exit
logout

[root@bob234hwxworker5 ~]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
No errors were found.
ERROR: Exiting with exit code 1.


REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information.

2017-10-27 13:54:33,419 INFO - ******************************* Check database started *******************************
2017-10-27 13:54:37,112 INFO - Checking for configs not mapped to any cluster
2017-10-27 13:54:37,132 INFO - Checking for configs selected more than once
2017-10-27 13:54:37,133 INFO - Checking for hosts without state
2017-10-27 13:54:37,134 INFO - Checking host component states count equals host component desired states count
2017-10-27 13:54:37,135 INFO - Checking services and their configs
2017-10-27 13:54:38,728 INFO - ******************************* Check database completed *******************************
2017-10-27 13:54:45,971 INFO - Checking DB store version
2017-10-27 13:54:46,543 INFO - DB store version is compatible
2017-10-27 14:03:11,066 INFO - ******************************* Check database started *******************************
2017-10-27 14:03:16,100 INFO - Checking for configs not mapped to any cluster
2017-10-27 14:03:16,111 ERROR - Exception occurred during check for not mapped configs to cluster procedure:
org.postgresql.util.PSQLException: ERROR: relation "clusterconfig" does not exist
Position: 23
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:283)
at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkForNotMappedConfigsToCluster(DatabaseConsistencyCheckHelper.java:164)
at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:107)
2017-10-27 14:03:16,112 INFO - Checking for configs selected more than once
2017-10-27 14:03:16,113 ERROR - Exception occurred during check for config selected more than ones procedure:
org.postgresql.util.PSQLException: ERROR: relation "clusterconfigmapping" does not exist
Position: 43
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:283)
at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkForConfigsSelectedMoreThanOnce(DatabaseConsistencyCheckHelper.java:221)
at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:109)
2017-10-27 14:03:16,113 INFO - Checking for hosts without state
2017-10-27 14:03:16,114 ERROR - Exception occurred during check for host without state procedure:
org.postgresql.util.PSQLException: ERROR: relation "hosts" does not exist
Position: 23
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:283)
at org.apache.ambari.server.checks.DatabaseConsistencyCheckHelper.checkForHostsWithoutState(DatabaseConsistencyCheckHelper.java:277)
at org.apache.ambari.server.checks.DatabaseConsistencyChecker.main(DatabaseConsistencyChecker.java:111)
"/var/log/ambari-server/ambari-server-check-database.log" 1235L, 113374C

8 REPLIES 8

Super Mentor

@Olivia Smith

I am suspecting that the "ambari.sql" file that you dumped from working cluster might be having a different "Database Schema" or Database Name .

So can you please share the output of the following :

1). The output of the following command:

# head -n 15 "/bkup/ambari.sql"
# head -n 15 "/bkup/hwx/ambari.sql"

As i see that you exported the DB dump to a different file and Importing from a different file.

.

2). Also please share the output of the following command:

# grep 'jdbc' /etc/ambari-server/conf/ambari.properties

.

Explorer

From the original mgmt. server.

head -15 /bkup/ambari.sql
--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = off;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET escape_string_warning = off;

--
-- Name: ambari; Type: SCHEMA; Schema: -; Owner: ambari

grep 'jdbc' /etc/ambari-server/conf/ambari.properties
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

================================================

From the server ambari.sql was restored to.

head -15 /bkup/hwx/hwx/ambari.sql
--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = off;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET escape_string_warning = off;

--
-- Name: ambari; Type: SCHEMA; Schema: -; Owner: ambari
--


--

grep 'jdbc' /etc/ambari-server/conf/ambari.properties
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat

Explorer

What I am trying to do is restore the ambari onto another vm in the cluster. If the ambari server goes down. Can I install ambari on another node in the same cluster and restore the database and configuration files to it?

Super Mentor

@Olivia Smith

When you perform ambari-server setup to point to your Postgres Instalce then are you chooseing option[1] or Option [4] for the postgres?

# ambari-server setup

Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (4):

.

You should choose Option [4] Not the Embedded Postgres.... But the PostgreSQL if you are using the Postgres Instance on your own to setup the DB.

Explorer

I choose

[1] - PostgreSQL (Embedded)

Thank you

Super Mentor

@Olivia Smith

You will need to use Option [4]

[4]-PostgreSQL

.So please try using the Option 4 and then restart ambari server again.

Super Mentor

@Olivia Smith

Good to know that the issue is resolved.

Yes, you are doing right. After installing to a new node (host) in the cluster we need to edit the ambari-agent.ini file to point to the new Ambari Server FQDN (Hostname) followed by ambari-agent restart.

/etc/ambari-agent/conf/ambari-agent.ini

.

If your issue is resolved then it will be great if you can mark this thread as answered (By Clicking on the "Accept" button) so that it will be helpfulfor other HCC users to quickly browse who are facing the same issue.

Explorer

Many thanks for your help.

It is working now.

Can I please ask if what I am doing is ok.

I am restoring ambari server to a less used node in the cluster and pointing the

/etc/ambari-agent/conf/ambari-agent.ini

on the other nodes to the new ambari manager.. restarting the ambari-agent

; ;