@mike_bronson7    How may connections do you see in the Postgres Stats?   You can try running the following SQL query in your Postgres Db to see what is the output? https://www.postgresql.org/docs/9.2/monitoring-stats.html numbackends => Number of backends currently connected to this database.   SELECT sum(numbackends) FROM pg_stat_database;   ... View more

@mike_bronson7    - Something is really not right on the Host .where postgres is running ... may be some load. Have you checked if that host is running out of resources ? - Is this postgres Db shared by few other applications like Ranger/Hive/Oozie .etc - What kind of background jobs or other processes are running on this node?     Here we see that sometimes the DB is taking more than 3 - 12 seconds delay to respond as well as the "Temporary failure in name resolution" as well as "PSQLException" issue is occurring intermittently. -- 11 Seconds Gap --  Thu Jan 30 14:48:35 UTC 2020 Checking Connection - 29 ERROR: Unable to connect to the DB. Please check DB connection properties. org.postgresql.util.PSQLException: The connection attempt failed. Thu Jan 30 14:48:46 UTC 2020 Checking Connection - 30 Connected to DB Successfully!   -------- 6 Seconds Gap------- Thu Jan 30 14:50:10 UTC 2020 Checking Connection - 92 Connected to DB Successfully! Thu Jan 30 14:50:17 UTC 2020 Checking Connection - 93 Connected to DB Successfully!   -------- 6 Seconds Gap------- Thu Jan 30 14:51:01 UTC 2020 Checking Connection - 130 Connected to DB Successfully! Thu Jan 30 14:51:07 UTC 2020 Checking Connection - 131 Connected to DB Successfully!   -------- 12 Seconds Gap ------- Thu Jan 30 14:51:45 UTC 2020 Checking Connection - 158 hostname: Temporary failure in name resolution Connected to DB Successfully! Thu Jan 30 14:51:56 UTC 2020 Checking Connection - 159 Connected to DB Successfully!     -------- 6 Seconds Gap------- Thu Jan 30 14:53:55 UTC 2020 Checking Connection - 254 Connected to DB Successfully! Thu Jan 30 14:54:01 UTC 2020 Checking Connection - 255 Connected to DB Successfully!     ... View more

@mike_bronson7  I am still suspecting that your Postgres is not returning connection consistently... I mean it might be either responding very slow or sometimes it might not be responding connection.....  (May be due to some LOAD on the system ..etc)   So can you please try running the following kind of script to check the DB connection consistency in a for loop for some time ?  Following is a smaple script "test_db_connection.sh" to test the connection consistently for 300 seconds # cat /tmp/test_db_connection.sh for i in {1..300} do echo `date` " Checking Connection - $i" /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver sleep 1 done . Please run the script as following and monitor if we are continuously getting the connection? # chmod 755 /tmp/test_db_connection.sh /tmp/test_db_connection.sh Thu Jan 30 14:41:10 UTC 2020 Checking Connection - 1 Connected to DB Successfully! Thu Jan 30 14:41:11 UTC 2020 Checking Connection - 2 Connected to DB Successfully! Thu Jan 30 14:41:13 UTC 2020 Checking Connection - 3 Connected to DB Successfully! . . ... View more

@mike_bronson7  What is the latest error in the "ambari-server.log" ? Also what is the output of the Test Connection command ? ... View more

@mike_bronson7  Usually Postgres writesinflight data and peristent data files inside the following dir " /var/lib/pgsql/data/pg_xlog"   # ls -lart /var/lib/pgsql/data/pg_xlog Do you see this file "000000010000003A00000023" there int he above output ...any permission issue .. Or has anyone deleted that file by mistake ?     So the previously mentioned error indicates that Postgres is not starting successfully because of that missing  file. FATAL: the database system is starting up LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory         ... View more

@mike_bronson7  Ambari is not listening on port 8080 because before starting to listen on that port it has to be able to connect to the DB successfully.   But in your case Postgres itself is not starting successfully that why we see these errors FATAL: the database system is starting up LOG: could not open file "pg_xlog/000000010000003A00000023" (log file 58, segment 35): No such file or directory   Please consult with your Postgres DB admin to get the Postgres fixed first it should ot be showing those FATAL errors (if those are the latest errors after postgres restart) then that needs to be fixed first.         ... View more

@mike_bronson7 Usually we find Postgres Db logs inside " /var/lib/pgsql/data/pg_log/" dir ... View more

@mike_bronson7  As you mentioned test Db command  " its hang ( not return prompt ) " This indicates some issue from Postgres side.   Hence can you please try restarting Postgres DB once . then re run the Test command again.   Still if you face any issue then please share the full Postgres log.   ... View more

@mike_bronson7  As you are using embedded Local postgres instance hence you can replace "localhost" with `hostname -f` like following. Or just use the output of "hostname -f" command in the previously mentioned command to replace "localhost"   # /usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-server/resources/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-42.2.2.jar org.apache.ambari.server.DBConnectionVerification jdbc:postgresql://`hostname -f`:5432/ambari ambari bigdata org.postgresql.Driver   . ... View more

@mike_bronson7  As mentioned earlier the PATH "/usr/lib/ambari-server/postgresql-42.2.2.jar" might be different in your casse.   So please check what is the JAR file name in your case. # ls -lart /usr/lib/ambari-server/postgresql-* ... View more