Support Questions

Find answers, ask questions, and share your expertise

postgresql + ambari server failed to open port 8080

avatar

We have ambari cluster

we have issue with ambari server

example

ambari-server start

Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start............................................................ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.

ambari server create the PID and service is up but server not listening on port 8080

we check the firewall and iptables but this services are down

we check the PostgreSQL service ( is running well )

systemctl status postgresql.service postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)   Active: active (running) since Thu 2020-01-30 12:49:26 UTC; 20min ago

we also checked the following

cat /etc/ambari-server/conf/password.dat
bigdata

grep 'ambari' /var/lib/pgsql/data/pg_hba.conf
local  all  ambari,mapred md5
host  all   ambari,mapred 0.0.0.0/0  md5
host  all   ambari,mapred ::/0 md5



grep 'jdbc'  /etc/ambari-server/conf/ambari.properties
custom.postgres.jdbc.name=postgresql-jdbc.jar
previous.custom.postgres.jdbc.name=postgresql-jdbc.jar
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=ambari
server.jdbc.postgres.schema=ambari
server.jdbc.user.name=ambari

but seems they are ok

so what could be the reason that the default port 8080 of ambari server isn’t opened

30 Jan 2020 12:51:50,152 ERROR [main] DBAccessorImpl:119 - Error while creating database accessor
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207)        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)        at org.postgresql.Driver.makeConnection(Driver.java:410)        at org.postgresql.Driver.connect(Driver.java:280)        at java.sql.DriverManager.getConnection(DriverManager.java:664)        at java.sql.DriverManager.getConnection(DriverManager.java:247)        at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:93)        at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>)        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110)        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94)        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)        at com.google.inject.Scopes$1$1.get(Scopes.java:65)        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)        at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110)        at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78)        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170)        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)        at com.google.inject.Guice.createInjector(Guice.java:95)        at com.google.inject.Guice.createInjector(Guice.java:72)        at com.google.inject.Guice.createInjector(Guice.java:62)        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1045)Caused by: java.net.ConnectException: Connection timed out (Connection timed out)        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:589)        at java.net.Socket.connect(Socket.java:538)        at org.postgresql.core.PGStream.<init>(PGStream.java:60)        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
        ... 52 more

 

Michael-Bronson
1 ACCEPTED SOLUTION

avatar

Dear Jay

 

finally we found the issue

 

it was about the mistake in /etc/hosts file

 

instead of 127.0.0.1 , it was the ip address - 27.0.0.1

 

so we fix it

 

and restart the postgresql and ambari

 

now all are fine 

 

 

Michael-Bronson

View solution in original post

32 REPLIES 32

avatar
Master Mentor

@mike_bronson7 

I am still suspecting that your Postgres is not returning connection consistently... I mean it might be either responding very slow or sometimes it might not be responding connection.....  (May be due to some LOAD on the system ..etc)

 

So can you please try running the following kind of script to check the DB connection consistency in a for loop for some time ?  Following is a smaple script "test_db_connection.sh" to test the connection consistently for 300 seconds

# cat /tmp/test_db_connection.sh

for i in {1..300}
do
   echo `date` " Checking Connection - $i"
   /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver

   sleep 1
done


.

Please run the script as following and monitor if we are continuously getting the connection?

# chmod 755 /tmp/test_db_connection.sh
 /tmp/test_db_connection.sh

Thu Jan 30 14:41:10 UTC 2020  Checking Connection - 1
Connected to DB Successfully!
Thu Jan 30 14:41:11 UTC 2020  Checking Connection - 2
Connected to DB Successfully!
Thu Jan 30 14:41:13 UTC 2020  Checking Connection - 3
Connected to DB Successfully!

.


.

avatar

Dear Jay

 

see please the attachment results 

 

 

bash /tmp/test_db_connection.sh
Thu Jan 30 14:47:51 UTC 2020 Checking Connection - 1
Connected to DB Successfully!
Thu Jan 30 14:47:52 UTC 2020 Checking Connection - 2
Connected to DB Successfully!
Thu Jan 30 14:47:58 UTC 2020 Checking Connection - 3
Connected to DB Successfully!
Thu Jan 30 14:48:00 UTC 2020 Checking Connection - 4
Connected to DB Successfully!
Thu Jan 30 14:48:01 UTC 2020 Checking Connection - 5
Connected to DB Successfully!
Thu Jan 30 14:48:02 UTC 2020 Checking Connection - 6
Connected to DB Successfully!
Thu Jan 30 14:48:03 UTC 2020 Checking Connection - 7
Connected to DB Successfully!
Thu Jan 30 14:48:04 UTC 2020 Checking Connection - 8
Connected to DB Successfully!
Thu Jan 30 14:48:06 UTC 2020 Checking Connection - 9
Connected to DB Successfully!
Thu Jan 30 14:48:07 UTC 2020 Checking Connection - 10
Connected to DB Successfully!
Thu Jan 30 14:48:08 UTC 2020 Checking Connection - 11
Connected to DB Successfully!
Thu Jan 30 14:48:09 UTC 2020 Checking Connection - 12
Connected to DB Successfully!
Thu Jan 30 14:48:10 UTC 2020 Checking Connection - 13
Connected to DB Successfully!
Thu Jan 30 14:48:12 UTC 2020 Checking Connection - 14
Connected to DB Successfully!
Thu Jan 30 14:48:13 UTC 2020 Checking Connection - 15
Connected to DB Successfully!
Thu Jan 30 14:48:14 UTC 2020 Checking Connection - 16
Connected to DB Successfully!
Thu Jan 30 14:48:15 UTC 2020 Checking Connection - 17
Connected to DB Successfully!
Thu Jan 30 14:48:16 UTC 2020 Checking Connection - 18
Connected to DB Successfully!
Thu Jan 30 14:48:18 UTC 2020 Checking Connection - 19
Connected to DB Successfully!
Thu Jan 30 14:48:19 UTC 2020 Checking Connection - 20
Connected to DB Successfully!
Thu Jan 30 14:48:20 UTC 2020 Checking Connection - 21
Connected to DB Successfully!
Thu Jan 30 14:48:21 UTC 2020 Checking Connection - 22
Connected to DB Successfully!
Thu Jan 30 14:48:22 UTC 2020 Checking Connection - 23
Connected to DB Successfully!
Thu Jan 30 14:48:24 UTC 2020 Checking Connection - 24
Connected to DB Successfully!
Thu Jan 30 14:48:25 UTC 2020 Checking Connection - 25
Connected to DB Successfully!
Thu Jan 30 14:48:26 UTC 2020 Checking Connection - 26
Connected to DB Successfully!
Thu Jan 30 14:48:27 UTC 2020 Checking Connection - 27
Connected to DB Successfully!
Thu Jan 30 14:48:33 UTC 2020 Checking Connection - 28
Connected to DB Successfully!
Thu Jan 30 14:48:35 UTC 2020 Checking Connection - 29
ERROR: Unable to connect to the DB. Please check DB connection properties.
org.postgresql.util.PSQLException: The connection attempt failed.
Thu Jan 30 14:48:46 UTC 2020 Checking Connection - 30
Connected to DB Successfully!
Thu Jan 30 14:48:47 UTC 2020 Checking Connection - 31
Connected to DB Successfully!
Thu Jan 30 14:48:48 UTC 2020 Checking Connection - 32
Connected to DB Successfully!
Thu Jan 30 14:48:49 UTC 2020 Checking Connection - 33
Connected to DB Successfully!
Thu Jan 30 14:48:51 UTC 2020 Checking Connection - 34
Connected to DB Successfully!
Thu Jan 30 14:48:52 UTC 2020 Checking Connection - 35
Connected to DB Successfully!
Thu Jan 30 14:48:53 UTC 2020 Checking Connection - 36
Connected to DB Successfully!
Thu Jan 30 14:48:54 UTC 2020 Checking Connection - 37
Connected to DB Successfully!
Thu Jan 30 14:48:55 UTC 2020 Checking Connection - 38
Connected to DB Successfully!
Thu Jan 30 14:48:57 UTC 2020 Checking Connection - 39
Connected to DB Successfully!
Thu Jan 30 14:48:58 UTC 2020 Checking Connection - 40
Connected to DB Successfully!
Thu Jan 30 14:48:59 UTC 2020 Checking Connection - 41
Connected to DB Successfully!
Thu Jan 30 14:49:00 UTC 2020 Checking Connection - 42
Connected to DB Successfully!
Thu Jan 30 14:49:01 UTC 2020 Checking Connection - 43
Connected to DB Successfully!
Thu Jan 30 14:49:03 UTC 2020 Checking Connection - 44
Connected to DB Successfully!
Thu Jan 30 14:49:04 UTC 2020 Checking Connection - 45
Connected to DB Successfully!
Thu Jan 30 14:49:05 UTC 2020 Checking Connection - 46
Connected to DB Successfully!
Thu Jan 30 14:49:06 UTC 2020 Checking Connection - 47
Connected to DB Successfully!
Thu Jan 30 14:49:07 UTC 2020 Checking Connection - 48
Connected to DB Successfully!
Thu Jan 30 14:49:09 UTC 2020 Checking Connection - 49
Connected to DB Successfully!
Thu Jan 30 14:49:10 UTC 2020 Checking Connection - 50
Connected to DB Successfully!
Thu Jan 30 14:49:11 UTC 2020 Checking Connection - 51
Connected to DB Successfully!
Thu Jan 30 14:49:12 UTC 2020 Checking Connection - 52
Connected to DB Successfully!
Thu Jan 30 14:49:23 UTC 2020 Checking Connection - 53
Connected to DB Successfully!
Thu Jan 30 14:49:25 UTC 2020 Checking Connection - 54
Connected to DB Successfully!
Thu Jan 30 14:49:26 UTC 2020 Checking Connection - 55
Connected to DB Successfully!
Thu Jan 30 14:49:27 UTC 2020 Checking Connection - 56
Connected to DB Successfully!
Thu Jan 30 14:49:28 UTC 2020 Checking Connection - 57
Connected to DB Successfully!
Thu Jan 30 14:49:29 UTC 2020 Checking Connection - 58
Connected to DB Successfully!
Thu Jan 30 14:49:31 UTC 2020 Checking Connection - 59
Connected to DB Successfully!
Thu Jan 30 14:49:32 UTC 2020 Checking Connection - 60
Connected to DB Successfully!
Thu Jan 30 14:49:33 UTC 2020 Checking Connection - 61
Connected to DB Successfully!
Thu Jan 30 14:49:34 UTC 2020 Checking Connection - 62
Connected to DB Successfully!
Thu Jan 30 14:49:35 UTC 2020 Checking Connection - 63
Connected to DB Successfully!
Thu Jan 30 14:49:37 UTC 2020 Checking Connection - 64
Connected to DB Successfully!
Thu Jan 30 14:49:38 UTC 2020 Checking Connection - 65
Connected to DB Successfully!
Thu Jan 30 14:49:39 UTC 2020 Checking Connection - 66
Connected to DB Successfully!
Thu Jan 30 14:49:40 UTC 2020 Checking Connection - 67
Connected to DB Successfully!
Thu Jan 30 14:49:41 UTC 2020 Checking Connection - 68
Connected to DB Successfully!
Thu Jan 30 14:49:43 UTC 2020 Checking Connection - 69
Connected to DB Successfully!
Thu Jan 30 14:49:44 UTC 2020 Checking Connection - 70
Connected to DB Successfully!
Thu Jan 30 14:49:45 UTC 2020 Checking Connection - 71
Connected to DB Successfully!
Thu Jan 30 14:49:46 UTC 2020 Checking Connection - 72
Connected to DB Successfully!
Thu Jan 30 14:49:47 UTC 2020 Checking Connection - 73
Connected to DB Successfully!
Thu Jan 30 14:49:49 UTC 2020 Checking Connection - 74
Connected to DB Successfully!
Thu Jan 30 14:49:50 UTC 2020 Checking Connection - 75
Connected to DB Successfully!
Thu Jan 30 14:49:51 UTC 2020 Checking Connection - 76
Connected to DB Successfully!
Thu Jan 30 14:49:52 UTC 2020 Checking Connection - 77
Connected to DB Successfully!
Thu Jan 30 14:49:53 UTC 2020 Checking Connection - 78
Connected to DB Successfully!
Thu Jan 30 14:49:55 UTC 2020 Checking Connection - 79
Connected to DB Successfully!
Thu Jan 30 14:49:56 UTC 2020 Checking Connection - 80
Connected to DB Successfully!
Thu Jan 30 14:49:57 UTC 2020 Checking Connection - 81
Connected to DB Successfully!
Thu Jan 30 14:49:58 UTC 2020 Checking Connection - 82
Connected to DB Successfully!
Thu Jan 30 14:49:59 UTC 2020 Checking Connection - 83
Connected to DB Successfully!
Thu Jan 30 14:50:01 UTC 2020 Checking Connection - 84
Connected to DB Successfully!
Thu Jan 30 14:50:02 UTC 2020 Checking Connection - 85
Connected to DB Successfully!
Thu Jan 30 14:50:03 UTC 2020 Checking Connection - 86
Connected to DB Successfully!
Thu Jan 30 14:50:04 UTC 2020 Checking Connection - 87
Connected to DB Successfully!
Thu Jan 30 14:50:05 UTC 2020 Checking Connection - 88
Connected to DB Successfully!
Thu Jan 30 14:50:07 UTC 2020 Checking Connection - 89
Connected to DB Successfully!
Thu Jan 30 14:50:08 UTC 2020 Checking Connection - 90
Connected to DB Successfully!
Thu Jan 30 14:50:09 UTC 2020 Checking Connection - 91
Connected to DB Successfully!
Thu Jan 30 14:50:10 UTC 2020 Checking Connection - 92
Connected to DB Successfully!
Thu Jan 30 14:50:17 UTC 2020 Checking Connection - 93
Connected to DB Successfully!
Thu Jan 30 14:50:18 UTC 2020 Checking Connection - 94
Connected to DB Successfully!
Thu Jan 30 14:50:19 UTC 2020 Checking Connection - 95
Connected to DB Successfully!
Thu Jan 30 14:50:20 UTC 2020 Checking Connection - 96
Connected to DB Successfully!
Thu Jan 30 14:50:21 UTC 2020 Checking Connection - 97
Connected to DB Successfully!
Thu Jan 30 14:50:23 UTC 2020 Checking Connection - 98
Connected to DB Successfully!
Thu Jan 30 14:50:24 UTC 2020 Checking Connection - 99
Connected to DB Successfully!
Thu Jan 30 14:50:25 UTC 2020 Checking Connection - 100
Connected to DB Successfully!
Thu Jan 30 14:50:26 UTC 2020 Checking Connection - 101
Connected to DB Successfully!
Thu Jan 30 14:50:27 UTC 2020 Checking Connection - 102
Connected to DB Successfully!
Thu Jan 30 14:50:29 UTC 2020 Checking Connection - 103
Connected to DB Successfully!
Thu Jan 30 14:50:30 UTC 2020 Checking Connection - 104
Connected to DB Successfully!
Thu Jan 30 14:50:31 UTC 2020 Checking Connection - 105
Connected to DB Successfully!
Thu Jan 30 14:50:32 UTC 2020 Checking Connection - 106
Connected to DB Successfully!
Thu Jan 30 14:50:33 UTC 2020 Checking Connection - 107
Connected to DB Successfully!
Thu Jan 30 14:50:35 UTC 2020 Checking Connection - 108
Connected to DB Successfully!
Thu Jan 30 14:50:36 UTC 2020 Checking Connection - 109
Connected to DB Successfully!
Thu Jan 30 14:50:37 UTC 2020 Checking Connection - 110
Connected to DB Successfully!
Thu Jan 30 14:50:38 UTC 2020 Checking Connection - 111
Connected to DB Successfully!
Thu Jan 30 14:50:39 UTC 2020 Checking Connection - 112
Connected to DB Successfully!
Thu Jan 30 14:50:41 UTC 2020 Checking Connection - 113
Connected to DB Successfully!
Thu Jan 30 14:50:42 UTC 2020 Checking Connection - 114
Connected to DB Successfully!
Thu Jan 30 14:50:43 UTC 2020 Checking Connection - 115
Connected to DB Successfully!
Thu Jan 30 14:50:44 UTC 2020 Checking Connection - 116
Connected to DB Successfully!
Thu Jan 30 14:50:45 UTC 2020 Checking Connection - 117
Connected to DB Successfully!
Thu Jan 30 14:50:47 UTC 2020 Checking Connection - 118
Connected to DB Successfully!
Thu Jan 30 14:50:48 UTC 2020 Checking Connection - 119
Connected to DB Successfully!
Thu Jan 30 14:50:49 UTC 2020 Checking Connection - 120
Connected to DB Successfully!
Thu Jan 30 14:50:50 UTC 2020 Checking Connection - 121
Connected to DB Successfully!
Thu Jan 30 14:50:51 UTC 2020 Checking Connection - 122
Connected to DB Successfully!
Thu Jan 30 14:50:52 UTC 2020 Checking Connection - 123
Connected to DB Successfully!
Thu Jan 30 14:50:54 UTC 2020 Checking Connection - 124
Connected to DB Successfully!
Thu Jan 30 14:50:55 UTC 2020 Checking Connection - 125
Connected to DB Successfully!
Thu Jan 30 14:50:56 UTC 2020 Checking Connection - 126
Connected to DB Successfully!
Thu Jan 30 14:50:57 UTC 2020 Checking Connection - 127
Connected to DB Successfully!
Thu Jan 30 14:50:59 UTC 2020 Checking Connection - 128
Connected to DB Successfully!
Thu Jan 30 14:51:00 UTC 2020 Checking Connection - 129
Connected to DB Successfully!
Thu Jan 30 14:51:01 UTC 2020 Checking Connection - 130
Connected to DB Successfully!
Thu Jan 30 14:51:07 UTC 2020 Checking Connection - 131
Connected to DB Successfully!
Thu Jan 30 14:51:08 UTC 2020 Checking Connection - 132
Connected to DB Successfully!
Thu Jan 30 14:51:10 UTC 2020 Checking Connection - 133
Connected to DB Successfully!
Thu Jan 30 14:51:11 UTC 2020 Checking Connection - 134
Connected to DB Successfully!
Thu Jan 30 14:51:12 UTC 2020 Checking Connection - 135
Connected to DB Successfully!
Thu Jan 30 14:51:13 UTC 2020 Checking Connection - 136
Connected to DB Successfully!
Thu Jan 30 14:51:14 UTC 2020 Checking Connection - 137
Connected to DB Successfully!
Thu Jan 30 14:51:16 UTC 2020 Checking Connection - 138
Connected to DB Successfully!
Thu Jan 30 14:51:17 UTC 2020 Checking Connection - 139
Connected to DB Successfully!
Thu Jan 30 14:51:18 UTC 2020 Checking Connection - 140
Connected to DB Successfully!
Thu Jan 30 14:51:19 UTC 2020 Checking Connection - 141
Connected to DB Successfully!
Thu Jan 30 14:51:20 UTC 2020 Checking Connection - 142
Connected to DB Successfully!
Thu Jan 30 14:51:22 UTC 2020 Checking Connection - 143
Connected to DB Successfully!
Thu Jan 30 14:51:28 UTC 2020 Checking Connection - 144
Connected to DB Successfully!
Thu Jan 30 14:51:29 UTC 2020 Checking Connection - 145
Connected to DB Successfully!
Thu Jan 30 14:51:30 UTC 2020 Checking Connection - 146
Connected to DB Successfully!
Thu Jan 30 14:51:32 UTC 2020 Checking Connection - 147
Connected to DB Successfully!
Thu Jan 30 14:51:33 UTC 2020 Checking Connection - 148
Connected to DB Successfully!
Thu Jan 30 14:51:34 UTC 2020 Checking Connection - 149
Connected to DB Successfully!
Thu Jan 30 14:51:35 UTC 2020 Checking Connection - 150
Connected to DB Successfully!
Thu Jan 30 14:51:36 UTC 2020 Checking Connection - 151
Connected to DB Successfully!
Thu Jan 30 14:51:38 UTC 2020 Checking Connection - 152
Connected to DB Successfully!
Thu Jan 30 14:51:39 UTC 2020 Checking Connection - 153
Connected to DB Successfully!
Thu Jan 30 14:51:40 UTC 2020 Checking Connection - 154
Connected to DB Successfully!
Thu Jan 30 14:51:41 UTC 2020 Checking Connection - 155
Connected to DB Successfully!
Thu Jan 30 14:51:42 UTC 2020 Checking Connection - 156
Connected to DB Successfully!
Thu Jan 30 14:51:44 UTC 2020 Checking Connection - 157
Connected to DB Successfully!
Thu Jan 30 14:51:45 UTC 2020 Checking Connection - 158
hostname: Temporary failure in name resolution
Connected to DB Successfully!
Thu Jan 30 14:51:56 UTC 2020 Checking Connection - 159
Connected to DB Successfully!
Thu Jan 30 14:51:57 UTC 2020 Checking Connection - 160
Connected to DB Successfully!
Thu Jan 30 14:51:58 UTC 2020 Checking Connection - 161
Connected to DB Successfully!
Thu Jan 30 14:52:00 UTC 2020 Checking Connection - 162
Connected to DB Successfully!
Thu Jan 30 14:52:01 UTC 2020 Checking Connection - 163
Connected to DB Successfully!
Thu Jan 30 14:52:02 UTC 2020 Checking Connection - 164
Connected to DB Successfully!
Thu Jan 30 14:52:03 UTC 2020 Checking Connection - 165
Connected to DB Successfully!
Thu Jan 30 14:52:04 UTC 2020 Checking Connection - 166
Connected to DB Successfully!
Thu Jan 30 14:52:06 UTC 2020 Checking Connection - 167
Connected to DB Successfully!
Thu Jan 30 14:52:07 UTC 2020 Checking Connection - 168
Connected to DB Successfully!
Thu Jan 30 14:52:08 UTC 2020 Checking Connection - 169
Connected to DB Successfully!
Thu Jan 30 14:52:09 UTC 2020 Checking Connection - 170
Connected to DB Successfully!
Thu Jan 30 14:52:10 UTC 2020 Checking Connection - 171
Connected to DB Successfully!
Thu Jan 30 14:52:12 UTC 2020 Checking Connection - 172
Connected to DB Successfully!
Thu Jan 30 14:52:13 UTC 2020 Checking Connection - 173
Connected to DB Successfully!
Thu Jan 30 14:52:14 UTC 2020 Checking Connection - 174
Connected to DB Successfully!
Thu Jan 30 14:52:15 UTC 2020 Checking Connection - 175
Connected to DB Successfully!
Thu Jan 30 14:52:16 UTC 2020 Checking Connection - 176
Connected to DB Successfully!
Thu Jan 30 14:52:18 UTC 2020 Checking Connection - 177
Connected to DB Successfully!
Thu Jan 30 14:52:19 UTC 2020 Checking Connection - 178
Connected to DB Successfully!
Thu Jan 30 14:52:20 UTC 2020 Checking Connection - 179
Connected to DB Successfully!
Thu Jan 30 14:52:21 UTC 2020 Checking Connection - 180
Connected to DB Successfully!
Thu Jan 30 14:52:22 UTC 2020 Checking Connection - 181
Connected to DB Successfully!
Thu Jan 30 14:52:24 UTC 2020 Checking Connection - 182
Connected to DB Successfully!
Thu Jan 30 14:52:25 UTC 2020 Checking Connection - 183
Connected to DB Successfully!
Thu Jan 30 14:52:26 UTC 2020 Checking Connection - 184
Connected to DB Successfully!
Thu Jan 30 14:52:27 UTC 2020 Checking Connection - 185
Connected to DB Successfully!
Thu Jan 30 14:52:28 UTC 2020 Checking Connection - 186
Connected to DB Successfully!
Thu Jan 30 14:52:30 UTC 2020 Checking Connection - 187
Connected to DB Successfully!
Thu Jan 30 14:52:36 UTC 2020 Checking Connection - 188
Connected to DB Successfully!
Thu Jan 30 14:52:37 UTC 2020 Checking Connection - 189
Connected to DB Successfully!
Thu Jan 30 14:52:38 UTC 2020 Checking Connection - 190
Connected to DB Successfully!
Thu Jan 30 14:52:40 UTC 2020 Checking Connection - 191
Connected to DB Successfully!
Thu Jan 30 14:52:41 UTC 2020 Checking Connection - 192
Connected to DB Successfully!
Thu Jan 30 14:52:42 UTC 2020 Checking Connection - 193
Connected to DB Successfully!
Thu Jan 30 14:52:43 UTC 2020 Checking Connection - 194
Connected to DB Successfully!
Thu Jan 30 14:52:44 UTC 2020 Checking Connection - 195
Connected to DB Successfully!
Thu Jan 30 14:52:45 UTC 2020 Checking Connection - 196
Connected to DB Successfully!
Thu Jan 30 14:52:47 UTC 2020 Checking Connection - 197
Connected to DB Successfully!
Thu Jan 30 14:52:48 UTC 2020 Checking Connection - 198
Connected to DB Successfully!
Thu Jan 30 14:52:49 UTC 2020 Checking Connection - 199
Connected to DB Successfully!
Thu Jan 30 14:52:50 UTC 2020 Checking Connection - 200
Connected to DB Successfully!
Thu Jan 30 14:52:51 UTC 2020 Checking Connection - 201
Connected to DB Successfully!
Thu Jan 30 14:52:53 UTC 2020 Checking Connection - 202
Connected to DB Successfully!
Thu Jan 30 14:52:54 UTC 2020 Checking Connection - 203
Connected to DB Successfully!
Thu Jan 30 14:52:55 UTC 2020 Checking Connection - 204
Connected to DB Successfully!
Thu Jan 30 14:52:56 UTC 2020 Checking Connection - 205
Connected to DB Successfully!
Thu Jan 30 14:52:58 UTC 2020 Checking Connection - 206
Connected to DB Successfully!
Thu Jan 30 14:52:59 UTC 2020 Checking Connection - 207
Connected to DB Successfully!
Thu Jan 30 14:53:00 UTC 2020 Checking Connection - 208
Connected to DB Successfully!
Thu Jan 30 14:53:01 UTC 2020 Checking Connection - 209
Connected to DB Successfully!
Thu Jan 30 14:53:02 UTC 2020 Checking Connection - 210
Connected to DB Successfully!
Thu Jan 30 14:53:04 UTC 2020 Checking Connection - 211
Connected to DB Successfully!
Thu Jan 30 14:53:05 UTC 2020 Checking Connection - 212
Connected to DB Successfully!
Thu Jan 30 14:53:06 UTC 2020 Checking Connection - 213
Connected to DB Successfully!
Thu Jan 30 14:53:07 UTC 2020 Checking Connection - 214
Connected to DB Successfully!
Thu Jan 30 14:53:08 UTC 2020 Checking Connection - 215
Connected to DB Successfully!
Thu Jan 30 14:53:10 UTC 2020 Checking Connection - 216
Connected to DB Successfully!
Thu Jan 30 14:53:11 UTC 2020 Checking Connection - 217
Connected to DB Successfully!
Thu Jan 30 14:53:12 UTC 2020 Checking Connection - 218
Connected to DB Successfully!
Thu Jan 30 14:53:13 UTC 2020 Checking Connection - 219
Connected to DB Successfully!
Thu Jan 30 14:53:14 UTC 2020 Checking Connection - 220
Connected to DB Successfully!
Thu Jan 30 14:53:16 UTC 2020 Checking Connection - 221
Connected to DB Successfully!
Thu Jan 30 14:53:17 UTC 2020 Checking Connection - 222
Connected to DB Successfully!
Thu Jan 30 14:53:18 UTC 2020 Checking Connection - 223
Connected to DB Successfully!
Thu Jan 30 14:53:19 UTC 2020 Checking Connection - 224
Connected to DB Successfully!
Thu Jan 30 14:53:20 UTC 2020 Checking Connection - 225
Connected to DB Successfully!
Thu Jan 30 14:53:22 UTC 2020 Checking Connection - 226
Connected to DB Successfully!
Thu Jan 30 14:53:23 UTC 2020 Checking Connection - 227
Connected to DB Successfully!
Thu Jan 30 14:53:24 UTC 2020 Checking Connection - 228
Connected to DB Successfully!
Thu Jan 30 14:53:25 UTC 2020 Checking Connection - 229
Connected to DB Successfully!
Thu Jan 30 14:53:26 UTC 2020 Checking Connection - 230
Connected to DB Successfully!
Thu Jan 30 14:53:28 UTC 2020 Checking Connection - 231
Connected to DB Successfully!
Thu Jan 30 14:53:29 UTC 2020 Checking Connection - 232
Connected to DB Successfully!
Thu Jan 30 14:53:30 UTC 2020 Checking Connection - 233
Connected to DB Successfully!
Thu Jan 30 14:53:31 UTC 2020 Checking Connection - 234
Connected to DB Successfully!
Thu Jan 30 14:53:32 UTC 2020 Checking Connection - 235
Connected to DB Successfully!
Thu Jan 30 14:53:34 UTC 2020 Checking Connection - 236
Connected to DB Successfully!
Thu Jan 30 14:53:35 UTC 2020 Checking Connection - 237
Connected to DB Successfully!
Thu Jan 30 14:53:36 UTC 2020 Checking Connection - 238
Connected to DB Successfully!
Thu Jan 30 14:53:37 UTC 2020 Checking Connection - 239
Connected to DB Successfully!
Thu Jan 30 14:53:38 UTC 2020 Checking Connection - 240
Connected to DB Successfully!
Thu Jan 30 14:53:40 UTC 2020 Checking Connection - 241
Connected to DB Successfully!
Thu Jan 30 14:53:41 UTC 2020 Checking Connection - 242
Connected to DB Successfully!
Thu Jan 30 14:53:42 UTC 2020 Checking Connection - 243
Connected to DB Successfully!
Thu Jan 30 14:53:43 UTC 2020 Checking Connection - 244
Connected to DB Successfully!
Thu Jan 30 14:53:44 UTC 2020 Checking Connection - 245
Connected to DB Successfully!
Thu Jan 30 14:53:46 UTC 2020 Checking Connection - 246
Connected to DB Successfully!
Thu Jan 30 14:53:47 UTC 2020 Checking Connection - 247
Connected to DB Successfully!
Thu Jan 30 14:53:48 UTC 2020 Checking Connection - 248
Connected to DB Successfully!
Thu Jan 30 14:53:49 UTC 2020 Checking Connection - 249
Connected to DB Successfully!
Thu Jan 30 14:53:50 UTC 2020 Checking Connection - 250
Connected to DB Successfully!
Thu Jan 30 14:53:52 UTC 2020 Checking Connection - 251
Connected to DB Successfully!
Thu Jan 30 14:53:53 UTC 2020 Checking Connection - 252
Connected to DB Successfully!
Thu Jan 30 14:53:54 UTC 2020 Checking Connection - 253
Connected to DB Successfully!
Thu Jan 30 14:53:55 UTC 2020 Checking Connection - 254
Connected to DB Successfully!
Thu Jan 30 14:54:01 UTC 2020 Checking Connection - 255
Connected to DB Successfully!
Thu Jan 30 14:54:03 UTC 2020 Checking Connection - 256
Connected to DB Successfully!
Thu Jan 30 14:54:09 UTC 2020 Checking Connection - 257
Connected to DB Successfully!
Thu Jan 30 14:54:10 UTC 2020 Checking Connection - 258
Connected to DB Successfully!
Thu Jan 30 14:54:11 UTC 2020 Checking Connection - 259
Connected to DB Successfully!
Thu Jan 30 14:54:12 UTC 2020 Checking Connection - 260
Connected to DB Successfully!
Thu Jan 30 14:54:14 UTC 2020 Checking Connection - 261
Connected to DB Successfully!
Thu Jan 30 14:54:15 UTC 2020 Checking Connection - 262
hostname: Temporary failure in name resolution
Connected to DB Successfully!
Thu Jan 30 14:54:31 UTC 2020 Checking Connection - 263
Connected to DB Successfully!
Thu Jan 30 14:54:32 UTC 2020 Checking Connection - 264
Connected to DB Successfully!
Thu Jan 30 14:54:33 UTC 2020 Checking Connection - 265
Connected to DB Successfully!
Thu Jan 30 14:54:35 UTC 2020 Checking Connection - 266
Connected to DB Successfully!
Thu Jan 30 14:54:36 UTC 2020 Checking Connection - 267
Connected to DB Successfully!
Thu Jan 30 14:54:37 UTC 2020 Checking Connection - 268
Connected to DB Successfully!
Thu Jan 30 14:54:38 UTC 2020 Checking Connection - 269
Connected to DB Successfully!
Thu Jan 30 14:54:39 UTC 2020 Checking Connection - 270
Connected to DB Successfully!
Thu Jan 30 14:54:41 UTC 2020 Checking Connection - 271
Connected to DB Successfully!
Thu Jan 30 14:54:42 UTC 2020 Checking Connection - 272
Connected to DB Successfully!
Thu Jan 30 14:54:43 UTC 2020 Checking Connection - 273
Connected to DB Successfully!
Thu Jan 30 14:54:44 UTC 2020 Checking Connection - 274
Connected to DB Successfully!
Thu Jan 30 14:54:45 UTC 2020 Checking Connection - 275
Connected to DB Successfully!
Thu Jan 30 14:54:47 UTC 2020 Checking Connection - 276
Connected to DB Successfully!
Thu Jan 30 14:54:48 UTC 2020 Checking Connection - 277
Connected to DB Successfully!
Thu Jan 30 14:54:49 UTC 2020 Checking Connection - 278
Connected to DB Successfully!
Thu Jan 30 14:54:50 UTC 2020 Checking Connection - 279
Connected to DB Successfully!
Thu Jan 30 14:54:51 UTC 2020 Checking Connection - 280
Connected to DB Successfully!
Thu Jan 30 14:54:53 UTC 2020 Checking Connection - 281
Connected to DB Successfully!
Thu Jan 30 14:54:54 UTC 2020 Checking Connection - 282
Connected to DB Successfully!
Thu Jan 30 14:54:55 UTC 2020 Checking Connection - 283
Connected to DB Successfully!
Thu Jan 30 14:54:56 UTC 2020 Checking Connection - 284
Connected to DB Successfully!
Thu Jan 30 14:54:57 UTC 2020 Checking Connection - 285
Connected to DB Successfully!
Thu Jan 30 14:54:59 UTC 2020 Checking Connection - 286
Connected to DB Successfully!
Thu Jan 30 14:55:00 UTC 2020 Checking Connection - 287
Connected to DB Successfully!
Thu Jan 30 14:55:01 UTC 2020 Checking Connection - 288
Connected to DB Successfully!
Thu Jan 30 14:55:02 UTC 2020 Checking Connection - 289
Connected to DB Successfully!
Thu Jan 30 14:55:03 UTC 2020 Checking Connection - 290
Connected to DB Successfully!
Thu Jan 30 14:55:05 UTC 2020 Checking Connection - 291
Connected to DB Successfully!
Thu Jan 30 14:55:06 UTC 2020 Checking Connection - 292
Connected to DB Successfully!
Thu Jan 30 14:55:07 UTC 2020 Checking Connection - 293
Connected to DB Successfully!
Thu Jan 30 14:55:08 UTC 2020 Checking Connection - 294
Connected to DB Successfully!
Thu Jan 30 14:55:09 UTC 2020 Checking Connection - 295
Connected to DB Successfully!
Thu Jan 30 14:55:11 UTC 2020 Checking Connection - 296
Connected to DB Successfully!
Thu Jan 30 14:55:12 UTC 2020 Checking Connection - 297
Connected to DB Successfully!
Thu Jan 30 14:55:13 UTC 2020 Checking Connection - 298
Connected to DB Successfully!
Thu Jan 30 14:55:14 UTC 2020 Checking Connection - 299
Connected to DB Successfully!
Thu Jan 30 14:55:15 UTC 2020 Checking Connection - 300
Connected to DB Successfully!

Michael-Bronson

avatar
Master Mentor

@mike_bronson7 

 

- Something is really not right on the Host .where postgres is running ... may be some load. Have you checked if that host is running out of resources ?

- Is this postgres Db shared by few other applications like Ranger/Hive/Oozie .etc
- What kind of background jobs or other processes are running on this node?

 

 

Here we see that sometimes the DB is taking more than 3 - 12 seconds delay to respond as well as the "Temporary failure in name resolution" as well as "PSQLException" issue is occurring intermittently.

-- 11 Seconds Gap -- 

Thu Jan 30 14:48:35 UTC 2020 Checking Connection - 29
ERROR: Unable to connect to the DB. Please check DB connection properties.
org.postgresql.util.PSQLException: The connection attempt failed.
Thu Jan 30 14:48:46 UTC 2020 Checking Connection - 30
Connected to DB Successfully!

 

-------- 6 Seconds Gap-------
Thu Jan 30 14:50:10 UTC 2020 Checking Connection - 92
Connected to DB Successfully!
Thu Jan 30 14:50:17 UTC 2020 Checking Connection - 93
Connected to DB Successfully!

 

-------- 6 Seconds Gap-------
Thu Jan 30 14:51:01 UTC 2020 Checking Connection - 130
Connected to DB Successfully!
Thu Jan 30 14:51:07 UTC 2020 Checking Connection - 131
Connected to DB Successfully!

 

-------- 12 Seconds Gap -------
Thu Jan 30 14:51:45 UTC 2020 Checking Connection - 158
hostname: Temporary failure in name resolution
Connected to DB Successfully!
Thu Jan 30 14:51:56 UTC 2020 Checking Connection - 159
Connected to DB Successfully!

 

 

-------- 6 Seconds Gap-------

Thu Jan 30 14:53:55 UTC 2020 Checking Connection - 254
Connected to DB Successfully!
Thu Jan 30 14:54:01 UTC 2020 Checking Connection - 255
Connected to DB Successfully!

 

 

avatar

hi Jay

 

no problem of memory / or cpu load average / or linux partitions / or any other resource  

Michael-Bronson

avatar

 

 

when we run

 

systemctl status postgresql.service

 

we see connection to HIVE and to airflow airflow_db

 

but this is as all clusters ( we have 58 ambari clusters with the same HIVE connection without  problems )

Michael-Bronson

avatar

hi Jay

 

 

what about other options - as ambari-server setup , or other solution ?

Michael-Bronson

avatar
Master Mentor

@mike_bronson7 

 

How may connections do you see in the Postgres Stats?

 

You can try running the following SQL query in your Postgres Db to see what is the output? https://www.postgresql.org/docs/9.2/monitoring-stats.html

numbackends => Number of backends currently connected to this database.

 

 SELECT sum(numbackends) FROM pg_stat_database;

 

avatar

psql ambari -U ambari
psql (9.2.13)
Type "help" for help.

ambari=> SELECT sum(numbackends) FROM pg_stat_database;
sum
-----
66
(1 row)

ambari=>

Michael-Bronson

avatar

Jay

 

 

is it possible to re-install ambari server? , without loosing data ?

 

 

 

Michael-Bronson

avatar

not clear to me how ambari server is up

 

but port 8080 not open , 

 

I am also try to use other port as 8899

 

but the same behavior , very strange 

Michael-Bronson