Cloudera Manager UI doesnt start, clouder-scm-server dead and pid file exists

[#1]

Hi All,

Clouder Manager UI doesnt start at all. Tried opening localhost:7180 or 127.0.0.1:7180, nothing worked.

Opening a new issue since the exception for cloudera-scm-server is different in my case and none of solutions in previous thread really matched. 

Cloudera manager worked till yesterday and I applied no update till today when its UI is not startng at all.

I dont understand why issue started today suddenly. 😞

I checked other threads in this community and did following steps in set of attempts, but clouder-scm-server still fails to keep started:-

1. Checked Cloudera-scm-server status:

[root@quickstart cloudera-scm-server]# service cloudera-scm-server status
cloudera-scm-server dead but pid file exists

2. Removed pid as

 rm /var/run/cloudera-scm-server.pid
3. Tried to stop cloudera-scm-server-db, but got issue!

[cloudera@quickstart ~]$ sudo service cloudera-scm-server-db stop
cloudera-scm-server-db: unrecognized service

4. One of the thread suggested that postgresql might not be running,so i checked in  /etc/rc.d/init.d/ for postgresql.

  But its not there!

  [root@quickstart ~]# service postgresql start
   postgresql: unrecognized service

5. I checked in /etc/hosts if loopback address is missing 127.0.0.1 localhost. But the entry is already there.

6. Checked if port 7180 is listening by command:

    netstat -na | grep 7180

    But I get no output here.

7. I checked /var/log/cloudera-scm-server/cloudera-scm-server.out for an issue, there I get below exception:-

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Exception in thread "MainThread" javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
	at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)
	at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:62)
	at com.cloudera.enterprise.AbstractWrappedEntityManager.beginForRollbackAndReadonly(AbstractWrappedEntityManager.java:89)
	at com.cloudera.cmf.persist.CmfEntityManager.beginForRollbackAndReadonly(CmfEntityManager.java:347)
	at com.cloudera.server.cmf.Main.initializeCustomQueryCache(Main.java:705)
	at com.cloudera.server.cmf.Main.<init>(Main.java:307)
	at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.hibernate.exception.JDBCConnectionException: Could not open connection
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
	at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)
	at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)
	at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
	at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
	at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1426)
	at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:59)
	... 5 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 108,228 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1036)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3427)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3327)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
	at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5133)
	at com.mchange.v2.c3p0.impl.NewProxyConnection.setTransactionIsolation(NewProxyConnection.java:701)
	at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:86)
	at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
	at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)
	... 10 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2914)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3337)
	... 20 more

I do not understand precise meaning of this exception and root cause for the issue.

NOTE: Internet is working on VM 

Please, please help suggest solution. 

[#2]

Hi @bgooley 

Thanks for the reply.

I checked the var/log/cloudera-scm-server.log. 

Sharing the exception part at the time when vm starts at below link:-

cloudera-scm-server.log

Sorry for the big log but posted all so that no important detail is missed.

Further, I again confirmed that cloudera-scm-server is not up.

It fails to keep up since the start of the issue.

I checked the process cmf.Main, it doesnt exist!

Since I am facing this issue, cannot work on VM really.

So since VM is the freshly started with this issue, so no scope for any long-running/hung queries cant be there!

This is verified here:-

mysql> SHOW FULL PROCESSLIST;
+----+------+-----------+------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------+------+---------+------+-------+-----------------------+
| 2 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST |
+----+------+-----------+------+---------+------+-------+-----------------------+
1 row in set (0.00 sec)

Please suggest, the steps for resolving the issue ..(any hint from cloudera-scm-server.log)

I am stuck with this issue since 10 days.:(

Thanks in advance.

[#3]

@manu_009,

Unfortunately, the CM log kind of shows us more of the same:  the server attempts to execute a query but communication with MySQL fails.

If this is a problem where the server cannot start, I'd be interested in seeing the logs from startup just to see if there are any clues when CM attempts to connect to MySQL initially.  If it gets by that part but then fails later on, that could tell us something.

Since the CM log shows it is waiting for packets that never arrive, that means it would be good to look at the network and MySQL to see if it gets connections from CM.

I'd recommend the following:

- Try starting CM and while CM is starting, run the "show full processlist;" to see if any connections are establishjed

You could also use netstat -nap or similar to see if connections are established.

- Check MySQL documentation... see if there is a way to debug incoming connections

- Try running this on the CM host before trying to start:  "tcpdump -i any -w ~/mysql.pcap port 3306" (or whatever port your MySQL is listining on)

This will capture all packets going out to and coming back from the MySQL server.

After CM does not start, use Ctrl-c to quit tcpdump and finalize the pcap file.   You can use WireShark to review communication with MySQL.

You could also run tcpdump on the MySQL host (if it is not the CM host) to find out how communication looks on the MySQL host side.

Generally, the errors you are seeing do not indicate a problem with CM itself; rather, CM cannot read information from the database.