Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Clouder Manager Timeout - Problem with PostgreSQL

avatar
Contributor

I haven't logged into the Clouder Manager web portal in a few weeks so I don't know for how long this has happened, but when I tried to log into it today via a web browser, it took a long time and responded with a Server Not Found.

 

I SSHed into the node with cloudera manager and ran the following commands, receiving the following responses:

 

curl localhost:7180

 

<head><meta http-equiv="refresh" content="0;url=/cmf/"></head>

 

wget localhost:7180

 

HTTP request sent, awaiting response... No data received.
Retrying.

 

I was able to "telnet 7180" successfully.

 

netstat -nltp | grep 7180

 

tcp        0      0 0.0.0.0:7180                0.0.0.0:*                   LISTEN      10875/java

 

Here is some sample output from /var/log/cloudera-scm/server/cloudera-scm-server.log:

 

2014-01-06 12:09:44,361  WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@7090b499:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5a670992 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

...

Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)

...

2014-01-06 12:26:34,367  INFO [591785153@scm-web-5346:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-SERVICEMONITOR-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:26:34,379  WARN [773346975@scm-web-5366:spi.SqlExceptionHelper@143] SQL Error: 0, SQLState: null
2014-01-06 12:26:34,388 ERROR [773346975@scm-web-5366:spi.SqlExceptionHelper@144] Connections could not be acquired from the underlying database!
2014-01-06 12:26:34,389  INFO [773346975@scm-web-5366:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-EVENTSERVER-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:27:03,384  WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@4995f35:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@255f13b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

 

 

Seeing there was a problem with PostgreSQL, issuing "psql" returned the following:

 

psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

 

Issuing "service postgresql status" returned:

 

postmaster is stopped

 

Issuing "service postgresql start" returned:

 

/var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first.
                                                           [FAILED]

 

Any ideas on why this occurred and what I may do to fix it?

 

Thank you.

 

2 ACCEPTED SOLUTIONS

avatar
Super Collaborator

Hi Matthew,

 

The postgres instance in question here may be in quick shutdown mode where it's honoring only existing connections but not servicing new ones. The postgres instance that you are probably using is 'cloudera-scm-server-db', not just postgresql. Check

 

# service cloudera-scm-server-db status 

 

# psql -U scm -p 7432

 

then provide the password as found in /etc/cloudera-scm-server/db.properties to login if desired.

View solution in original post

avatar
Contributor

Hi Smark; thanks for your reply.

 

service cloudera-scm-server-db status 

 

pg_ctl: no server running

 

psql -U scm -p 7432

 

psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.7432"?

service cloudera-scm-server-db start

DB initialization done.
waiting for server to start....... done
server started

 

service cloudera-scm-server-db status 

 

pg_ctl: server is running (PID: 24118)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

 

After turning on the server, it worked.

 

Thanks!

View solution in original post

3 REPLIES 3

avatar
Super Collaborator

Hi Matthew,

 

The postgres instance in question here may be in quick shutdown mode where it's honoring only existing connections but not servicing new ones. The postgres instance that you are probably using is 'cloudera-scm-server-db', not just postgresql. Check

 

# service cloudera-scm-server-db status 

 

# psql -U scm -p 7432

 

then provide the password as found in /etc/cloudera-scm-server/db.properties to login if desired.

avatar
Contributor

Hi Smark; thanks for your reply.

 

service cloudera-scm-server-db status 

 

pg_ctl: no server running

 

psql -U scm -p 7432

 

psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.7432"?

service cloudera-scm-server-db start

DB initialization done.
waiting for server to start....... done
server started

 

service cloudera-scm-server-db status 

 

pg_ctl: server is running (PID: 24118)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

 

After turning on the server, it worked.

 

Thanks!

avatar
Expert Contributor

hey smark , i am having Matthew's same problem but his fix didn't work with me 

 

when i tried 

service cloudera-scm-server-db start

the output was : 

 

Creating DB navms for role NAVIGATORMETASERVER
waiting for server to start.... done
server started
psql: could not connect to server: Connection timed out
Is the server running on host "localhost" and accepting
TCP/IP connections on port 7432?
waiting for server to shut down..... done
server stopped
Unable to create database role navms, giving up
waiting for server to start..... done
server started

 

its status : 

 

pg_ctl: server is running (PID: 3808)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

 

but when i tried to turn the server on :

service cloudera-scm-server start

 

it says server running [ok]

but when i get its status after a while i found it dead 

 

service cloudera-scm-server status
cloudera-scm-server dead but pid file exists

 

My log error is same as the above