question Re: Clouder Manager Timeout - Problem with PostgreSQL in Archives of Support Questions (Read Only)

Clouder Manager Timeout - Problem with PostgreSQL

MatthewMoisen — Fri, 16 Sep 2022 08:52:04 GMT

I haven't logged into the Clouder Manager web portal in a few weeks so I don't know for how long this has happened, but when I tried to log into it today via a web browser, it took a long time and responded with a Server Not Found.

I SSHed into the node with cloudera manager and ran the following commands, receiving the following responses:

curl localhost:7180

wget localhost:7180

HTTP request sent, awaiting response... No data received.
Retrying.

I was able to "telnet 7180" successfully.

netstat -nltp | grep 7180

tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 10875/java

Here is some sample output from /var/log/cloudera-scm/server/cloudera-scm-server.log:

2014-01-06 12:09:44,361 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@7090b499:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5a670992 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

...

Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)

...

2014-01-06 12:26:34,367 INFO [591785153@scm-web-5346:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-SERVICEMONITOR-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:26:34,379 WARN [773346975@scm-web-5366:spi.SqlExceptionHelper@143] SQL Error: 0, SQLState: null
2014-01-06 12:26:34,388 ERROR [773346975@scm-web-5366:spi.SqlExceptionHelper@144] Connections could not be acquired from the underlying database!
2014-01-06 12:26:34,389 INFO [773346975@scm-web-5366:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-EVENTSERVER-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:27:03,384 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@4995f35:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@255f13b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

Seeing there was a problem with PostgreSQL, issuing "psql" returned the following:

psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

Issuing "service postgresql status" returned:

postmaster is stopped

Issuing "service postgresql start" returned:

/var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first.
[FAILED]

Any ideas on why this occurred and what I may do to fix it?

Thank you.

Re: Clouder Manager Timeout - Problem with PostgreSQL

smark — Mon, 06 Jan 2014 22:21:56 GMT

Hi Matthew,

The postgres instance in question here may be in quick shutdown mode where it's honoring only existing connections but not servicing new ones. The postgres instance that you are probably using is 'cloudera-scm-server-db', not just postgresql. Check

# service cloudera-scm-server-db status

# psql -U scm -p 7432

then provide the password as found in /etc/cloudera-scm-server/db.properties to login if desired.

Re: Clouder Manager Timeout - Problem with PostgreSQL

MatthewMoisen — Tue, 07 Jan 2014 00:52:46 GMT

Hi Smark; thanks for your reply.

service cloudera-scm-server-db status

pg_ctl: no server running

psql -U scm -p 7432

psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.7432"?

service cloudera-scm-server-db start

DB initialization done.
waiting for server to start....... done
server started

service cloudera-scm-server-db status

pg_ctl: server is running (PID: 24118)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

After turning on the server, it worked.

Thanks!

Re: Clouder Manager Timeout - Problem with PostgreSQL

tarekabouzeid91 — Tue, 10 Mar 2015 13:10:49 GMT

hey smark , i am having Matthew's same problem but his fix didn't work with me

when i tried

service cloudera-scm-server-db start

the output was :

Creating DB navms for role NAVIGATORMETASERVER
waiting for server to start.... done
server started
psql: could not connect to server: Connection timed out
Is the server running on host "localhost" and accepting
TCP/IP connections on port 7432?
waiting for server to shut down..... done
server stopped
Unable to create database role navms, giving up
waiting for server to start..... done
server started

its status :

pg_ctl: server is running (PID: 3808)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

but when i tried to turn the server on :

service cloudera-scm-server start

it says server running [ok]

but when i get its status after a while i found it dead

service cloudera-scm-server status
cloudera-scm-server dead but pid file exists

My log error is same as the above