Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager UI doesnt start, clouder-scm-server dead and pid file exists

Highlighted

Re: Cloudera Manager UI doesnt start, clouder-scm-server dead and pid file exists

Explorer

Hi @bgooley 

 

Thanks for the reply.

I checked the var/log/cloudera-scm-server.log. 

Sharing the exception part at the time when vm starts at below link:-

cloudera-scm-server.log

Sorry for the big log but posted all so that no important detail is missed.

 

Further, I again confirmed that cloudera-scm-server is not up.

It fails to keep up since the start of the issue.

I checked the process cmf.Main, it doesnt exist!

Since I am facing this issue, cannot work on VM really.

So since VM is the freshly started with this issue, so no scope for any long-running/hung queries cant be there!

This is verified here:-

 

mysql> SHOW FULL PROCESSLIST;
+----+------+-----------+------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------+------+---------+------+-------+-----------------------+
| 2 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST |
+----+------+-----------+------+---------+------+-------+-----------------------+
1 row in set (0.00 sec)

 

Please suggest, the steps for resolving the issue ..(any hint from cloudera-scm-server.log)

I am stuck with this issue since 10 days.:(

Thanks in advance.

 

Re: Cloudera Manager UI doesnt start, clouder-scm-server dead and pid file exists

Super Guru

@manu_009,

 

Unfortunately, the CM log kind of shows us more of the same:  the server attempts to execute a query but communication with MySQL fails.

 

If this is a problem where the server cannot start, I'd be interested in seeing the logs from startup just to see if there are any clues when CM attempts to connect to MySQL initially.  If it gets by that part but then fails later on, that could tell us something.

 

Since the CM log shows it is waiting for packets that never arrive, that means it would be good to look at the network and MySQL to see if it gets connections from CM.

 

I'd recommend the following:

 

- Try starting CM and while CM is starting, run the "show full processlist;" to see if any connections are establishjed

You could also use netstat -nap or similar to see if connections are established.

- Check MySQL documentation... see if there is a way to debug incoming connections

- Try running this on the CM host before trying to start:  "tcpdump -i any -w ~/mysql.pcap port 3306" (or whatever port your MySQL is listining on)

 

This will capture all packets going out to and coming back from the MySQL server.

After CM does not start, use Ctrl-c to quit tcpdump and finalize the pcap file.   You can use WireShark to review communication with MySQL.

 

You could also run tcpdump on the MySQL host (if it is not the CM host) to find out how communication looks on the MySQL host side.

 

Generally, the errors you are seeing do not indicate a problem with CM itself; rather, CM cannot read information from the database.