postgresql + ambari server failed to open port 8080

[#1]

We have ambari cluster

we have issue with ambari server

example

ambari-server start

Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.

ambari server create the PID and service is up but server not listening on port 8080

we check the firewall and iptables but this services are down

we check the PostgreSQL service ( is running well )

systemctl status postgresql.service● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)   Active: active (running) since Thu 2020-01-30 12:49:26 UTC; 20min ago

we also checked the following

cat /etc/ambari-server/conf/password.dat
bigdata

grep 'ambari' /var/lib/pgsql/data/pg_hba.conf
local  all  ambari,mapred md5
host  all   ambari,mapred 0.0.0.0/0  md5
host  all   ambari,mapred ::/0 md5



grep 'jdbc'  /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-jdbc.jar
previous.custom.postgres.jdbc.name=postgresql-jdbc.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari

but seems they are ok

so what could be the reason that the default port 8080 of ambari server isn’t opened

30 Jan 2020 12:51:50,152 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207)        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)        at org.postgresql.Driver.makeConnection(Driver.java:410)        at org.postgresql.Driver.connect(Driver.java:280)        at java.sql.DriverManager.getConnection(DriverManager.java:664)        at java.sql.DriverManager.getConnection(DriverManager.java:247)        at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93)        at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>)        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78)        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170)        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)        at com.google.inject.Guice.createInjector(Guice.java:95)        at com.google.inject.Guice.createInjector(Guice.java:72)        at com.google.inject.Guice.createInjector(Guice.java:62)        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)Caused by: java.net.ConnectException: Connection timed out (Connection timed out)        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:589)        at java.net.Socket.connect(Socket.java:538)        at org.postgresql.core.PGStream.<init>(PGStream.java:60)        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
        ... 52 more

[#2]

@mike_bronson7

Usually we find Postgres Db logs inside "

/var/lib/pgsql/data/pg_log/" dir

[#3]

more *
LOG: database system was interrupted; last known up at 2020-01-22 23:31:37 UTC
LOG: redo starts at 39/C7000080
LOG: consistent recovery state reached at 39/C70474A8
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory
LOG: redo done at 3A/22036568
LOG: last completed transaction was at log time 2020-01-30 10:41:43.937312+00
FATAL: the database system is starting up
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress

[#4]

Dear jay

we restart the postgreSQL 3 times but ambari still failed on port 8080  ( include the setting ) 

server.startup.web.timeout=150

[#5]

@mike_bronson7 

Ambari is not listening on port 8080 because before starting to listen on that port it has to be able to connect to the DB successfully.

But in your case Postgres itself is not starting successfully that why we see these errors

FATAL: the database system is starting up
LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory

Please consult with your Postgres DB admin to get the Postgres fixed first it should ot be showing those FATAL errors (if those are the latest errors after postgres restart) then that needs to be fixed first.

[#6]

@mike_bronson7 
Usually Postgres writesinflight data and peristent data files inside the following dir "

/var/lib/pgsql/data/pg_xlog"

# ls -lart /var/lib/pgsql/data/pg_xlog

Do you see this file "000000010000003A00000023" there int he above output ...any permission issue .. Or has anyone deleted that file by mistake ?  

So the previously mentioned error indicates that Postgres is not starting successfully because of that missing  file.

FATAL: the database system is starting up
LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory

[#7]

yes I add permissions to the file

seems the permissions added successfully 

and the fresh  log is like that

LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down at 2020-01-30 14:23:23 UTC
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

[#8]

after I fixed the permissions

I restart the PostgreSQL and ambari server

but still get

Waiting for server start.................................ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.

[#9]

@mike_bronson7 

What is the latest error in the "ambari-server.log" ?

Also what is the output of the Test Connection command ?

[#10]

yes - the same problem 

ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

[#11]

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver

Connected to DB Successfully!