I haven't logged into the Clouder Manager web portal in a few weeks so I don't know for how long this has happened, but when I tried to log into it today via a web browser, it took a long time and responded with a Server Not Found.
I SSHed into the node with cloudera manager and ran the following commands, receiving the following responses:
curl localhost:7180
<head><meta http-equiv="refresh" content="0;url=/cmf/"></head>
wget localhost:7180
HTTP request sent, awaiting response... No data received.
Retrying.
I was able to "telnet 7180" successfully.
netstat -nltp | grep 7180
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 10875/java
Here is some sample output from /var/log/cloudera-scm/server/cloudera-scm-server.log:
2014-01-06 12:09:44,361 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@7090b499:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5a670992 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)
...
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
...
2014-01-06 12:26:34,367 INFO [591785153@scm-web-5346:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-SERVICEMONITOR-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:26:34,379 WARN [773346975@scm-web-5366:spi.SqlExceptionHelper@143] SQL Error: 0, SQLState: null
2014-01-06 12:26:34,388 ERROR [773346975@scm-web-5366:spi.SqlExceptionHelper@144] Connections could not be acquired from the underlying database!
2014-01-06 12:26:34,389 INFO [773346975@scm-web-5366:cmf.AuthenticationFailureEventListener@19] Authentication failure for user: __cloudera_internal_user__mgmt1-EVENTSERVER-7e2762fc6af4771300e7773daf64993b
2014-01-06 12:27:03,384 WARN [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@4995f35:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@255f13b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)
Seeing there was a problem with PostgreSQL, issuing "psql" returned the following:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Issuing "service postgresql status" returned:
postmaster is stopped
Issuing "service postgresql start" returned:
/var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first.
[FAILED]
Any ideas on why this occurred and what I may do to fix it?
Thank you.
