Ambari server fails to start

[#1]

Hi,

I downloaded the sandbox from Hortonworks site and installed it successfully. As the ambari server wasn't running automatically, I installed the ambari server through command line and tried to start the server.

I am receiving the following error in the server log. I did connect using the same user and performed select on table successfully that is mentioned in the log. Any help is much appreciated.

16 Feb 2017 18:35:13,319 INFO [main] AmbariServer:913 - Getting the controller 16 Feb 2017 18:35:15,883 ERROR [main] AmbariServer:927 - Failed to run the Ambari Server Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: org.postgresql.util.PSQLException: ERROR: could not read block 0 of relation base/16384/16953: read only 0 of 8192 bytes Error Code: 0 Call: SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?) bind => [1 parameter bound] Query: ReadObjectQuery(name="readMetainfoEntity" referenceClass=MetainfoEntity sql="SELECT "metainfo_key", "metainfo_value" FROM metainfo WHERE ("metainfo_key" = ?)") at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:684)

[root@sandbox ~]# psql -U ambari -d ambari;

Password for user ambari:

psql (8.4.20) Type "help" for help.

ambari=> SELECT "metainfo_key", "metainfo_value" FROM metainfo;

metainfo_key | metainfo_value

--------------+----------------

(0 rows)

[#2]

@Manish Anand "metainfo" table should contain the ambari version and the HDP/HDP-UTIL repository informations. In your case it seems to be missing.

But the error "ERROR: could not read block 0 of relation base/16384/16953: read only 0 of 8192 bytes Error Code:" indicates more of a Database issue, Not directly related to ambari.

If it is a test system then may be you should try to recover any old ambari DB dump if you already have it.

.

[#3]

@Jay SenSharma

Would it be possible for you to provide me the ambari version and the HDP/HDP-UTIL repository information so that I can insert it into "metainfo" and test again?

[#4]

@Manish Anand

Some default values you can find from the SQL files like:

# grep -A3 'INSERT INTO metainfo'  /var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql
INSERT INTO metainfo(metainfo_key, metainfo_value) VALUES
('version','2.4.2');
COMMIT;

.

However i till believe that there is a Database Corruption based on the error message "could not read block 0 of relation base/16384/16953: read only". So i guess even if you restore that particular table still you will be facing the same issue, As "metainfo" is one of the first table that ambari looks at during the start.... so later you might find the same issue with other tables as well.

I will suggest if you have any prior DB dump then try to recover it.

- My "metainfo" table entry might not help... but sharing to show how it looks like normally:

.

[#5]

@Jay SenSharma

I do agree to your point that database is corrupted. I tried performing the insert to metainfo and received the same error:

ambari=> INSERT INTO metainfo(metainfo_key, metainfo_value) VALUES ('version','2.4.0');

ERROR: could not read block 0 of relation base/16384/16953: read only 0 of 8192 bytes

How shall I proceed as I don't have any prior DB dump?

[#6]

@Manish Anand

Without having an earlier DB dump it is hard to recover the DB.

However it is good that we isolated that the issue is actually with the "Postgres", Not with ambari. Postgres forums might have some quick workarounds for such corruption scenarios to fix the DB.

From Ambari side i do not think we have much to do.

If your cluster is just a test cluster (with minimal configuration) then may be you can do recreate fresh cluster. Or if you have any other cluster at your end then may be you can try recovering the entries on "metainfo" (it might help if only the "metainfo" table is corrupted)

I will share if i find something from Postgres side.

[#7]

@Manish Anand

Also please see if you can recreate the index on metainfo table. Something like described in:

http://stackoverflow.com/questions/6895736/error-could-not-read-block-4707-of-relation-1663-16384-16...

[#8]

@Jay SenSharma

I don't see any index defined on the metainfo table. How about migrating to MySQL and performing the ambari setup? Any suggestion?

[#9 · Accepted solution]

@Manish Anand

In MySQL also you will have no data ... because you do not have a dump. (Hence using MySQL / Postgres both are same in this scenario)

The current cluster information is anyway lost due to postgres table/db corruption. In this case you can try running the following command to reset postgres ambari DB (it will clear all previous data)

ambari-server reset

.

[#10]

@Jay SenSharma

I have performed the ambari server reset. What shall I do next for configuring the Ambari server and running the cluster wizard.?

[root@sandbox ~]# ambari-server reset

Using python /usr/bin/python Resetting ambari-server **** WARNING **** You are about to reset and clear the Ambari Server database. This will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard.

Are you SURE you want to perform the reset [yes/no] (no)? yes

Confirm server reset [yes/no](no)? yes

Resetting the Server database... Connecting to local database...done.

WARNING: Non critical error in DDL, use --verbose for more information Ambari Server 'reset' completed with warnings.

[root@sandbox ~]#