question postgresql + ambari server failed to open port 8080 in Support Questions

postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:24:16 GMT

We have ambari cluster

we have issue with ambari server

example

ambari-server start

Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.

ambari server create the PID and service is up but server not listening on port 8080

we check the firewall and iptables but this services are down

we check the PostgreSQL service ( is running well )

systemctl status postgresql.service● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)   Active: active (running) since Thu 2020-01-30 12:49:26 UTC; 20min ago

we also checked the following

cat /etc/ambari-server/conf/password.dat
bigdata

grep 'ambari' /var/lib/pgsql/data/pg_hba.conf
local  all  ambari,mapred md5
host  all   ambari,mapred 0.0.0.0/0  md5
host  all   ambari,mapred ::/0 md5



grep 'jdbc'  /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-jdbc.jar
previous.custom.postgres.jdbc.name=postgresql-jdbc.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari

but seems they are ok

so what could be the reason that the default port 8080 of ambari server isn’t opened

30 Jan 2020 12:51:50,152 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207)        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)        at org.postgresql.Driver.makeConnection(Driver.java:410)        at org.postgresql.Driver.connect(Driver.java:280)        at java.sql.DriverManager.getConnection(DriverManager.java:664)        at java.sql.DriverManager.getConnection(DriverManager.java:247)        at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93)        at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>)        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78)        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170)        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)        at com.google.inject.Guice.createInjector(Guice.java:95)        at com.google.inject.Guice.createInjector(Guice.java:72)        at com.google.inject.Guice.createInjector(Guice.java:62)        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)Caused by: java.net.ConnectException: Connection timed out (Connection timed out)        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:589)        at java.net.Socket.connect(Socket.java:538)        at org.postgresql.core.PGStream.<init>(PGStream.java:60)        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
        ... 52 more

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 13:49:53 GMT

@mike_bronson7

Can you please try the following command to verify if Postgres is listening to * address and port is opened?

# ps -ef | grep pgsql | grep ^postgres # netstat -tnlpa | grep 5432 # netstat -tnlpa | grep $POSTGRES_PID

If above does not show the port is listening properly then we might need to check the postgres logs.

Next can you please check if the Following Test Postgres DB connection command is running without any issue on Ambari Server host?

# /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-42.2.2.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://localhost:5432/ambari ambari bigdata org.postgresql.Driver

The please try to run the same command using HOSTNAME of PostgresDB (instead of "localhost")

Please NOTE that the "/usr/lib/ambari-server/postgresql-42.2.2.jar" PATH and the JDK installation path might be different in your case so please update the command accordingly.

.
Also as we see the message :

REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.

So it will be also good to try setting the following and then try to restart ambari again properto to a little high value like 120 or 150 seconds as sometimes ambari takes little longer time to start. Default value is 50 seconds.

server.startup.web.timeout=150

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 13:53:01 GMT

Dear Jay

here the first test

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-42.2.2.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://localhost:5432/ambari ambari bigdata org.postgresql.Driver
ERROR: Unable to connect to the DB. Please check DB connection properties.
java.lang.ClassNotFoundException: org.postgresql.Driver

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 13:54:31 GMT

@mike_bronson7

As mentioned earlier the PATH "/usr/lib/ambari-server/postgresql-42.2.2.jar" might be different in your casse.

So please check what is the JAR file name in your case.

# ls -lart /usr/lib/ambari-server/postgresql-*

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 13:54:45 GMT

about the second test

how to find the HOSTNAME of PostgresDB ?

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 13:57:20 GMT

@mike_bronson7

As you are using embedded Local postgres instance hence you can replace "localhost" with `hostname -f` like following. Or just use the output of "hostname -f" command in the previously mentioned command to replace "localhost"

# /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-42.2.2.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 13:57:24 GMT

sorry for my mistake

the correct is

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://localhost:5432/ambari ambari bigdata org.postgresql.Driver

but its hang ( not return prompt )

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 13:59:13 GMT

@mike_bronson7

As you mentioned test Db command " its hang ( not return prompt ) " This indicates some issue from Postgres side.

Hence can you please try restarting Postgres DB once . then re run the Test command again.

Still if you face any issue then please share the full Postgres log.

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:00:57 GMT

now I see the correct syntax ( I use full FQDN )

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver
Connected to DB Successfully!

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:01:58 GMT

now I see the correct syntax ( I use full FQDN )

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver
Connected to DB Successfully!

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 14:02:08 GMT

@mike_bronson7

Usually we find Postgres Db logs inside "

/var/lib/pgsql/data/pg_log/" dir

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:04:23 GMT

more *
LOG: database system was interrupted; last known up at 2020-01-22 23:31:37 UTC
LOG: redo starts at 39/C7000080
LOG: consistent recovery state reached at 39/C70474A8
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory
LOG: redo done at 3A/22036568
LOG: last completed transaction was at log time 2020-01-30 10:41:43.937312+00
FATAL: the database system is starting up
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress
WARNING: there is already a transaction in progress

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:11:06 GMT

Dear jay

we restart the postgreSQL 3 times but ambari still failed on port 8080 ( include the setting )

server.startup.web.timeout=150

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 14:11:30 GMT

@mike_bronson7

Ambari is not listening on port 8080 because before starting to listen on that port it has to be able to connect to the DB successfully.

But in your case Postgres itself is not starting successfully that why we see these errors

FATAL: the database system is starting up LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory

Please consult with your Postgres DB admin to get the Postgres fixed first it should ot be showing those FATAL errors (if those are the latest errors after postgres restart) then that needs to be fixed first.

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 14:24:48 GMT

@mike_bronson7
Usually Postgres writesinflight data and peristent data files inside the following dir "

/var/lib/pgsql/data/pg_xlog"

# ls -lart /var/lib/pgsql/data/pg_xlog

Do you see this file "000000010000003A00000023" there int he above output ...any permission issue .. Or has anyone deleted that file by mistake ?

So the previously mentioned error indicates that Postgres is not starting successfully because of that missing file.

FATAL: the database system is starting up LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:29:33 GMT

yes I add permissions to the file

seems the permissions added successfully

and the fresh log is like that

LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down at 2020-01-30 14:23:23 UTC
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:32:17 GMT

after I fixed the permissions

I restart the PostgreSQL and ambari server

but still get

Waiting for server start.................................ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more information.

Re: postgresql + ambari server failed to open port 8080

jsensharma — Thu, 30 Jan 2020 14:33:37 GMT

@mike_bronson7

What is the latest error in the "ambari-server.log" ?

Also what is the output of the Test Connection command ?

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:37:03 GMT

yes - the same problem

ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

Re: postgresql + ambari server failed to open port 8080

mike_bronson7 — Thu, 30 Jan 2020 14:38:11 GMT

/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver

Connected to DB Successfully!