Cloudera management service and all services timeout

[#1]

Hello,

My cluster was working fine and I hadn't monitored it for a few days. Recently when I saw, the services were not displaying their health. I tried restarting the cloudera management service and it gave me the following error -

"Command timed out after 150 seconds".

When I tried restarting any service, I get the same error message - and none of the services are working. Where can I start debugging the issue? Where are the configuration files of cloudera management service stored? 

cloudera-scm-agent, cloudera-scm-server, cloudera-scm-server-db are all running

[#2]

Are the nodes able to contact each other? Try telnet to port 9000 (CM Agent's port) from one host to another. Ping nodes from each other, ensure all of them can talk to the other. If the agents are running, you should be able to connect via telnet. The easiest action to take right now is to restart the agents on the nodes. Perform this on just one or a few nodes and try restarting services on them.

# service cloudera-scm-agent hard_restart  (this will restart any running services as well)

[#3]

Thanks Gautam for the quick reply.

I am able to telnet on port 9000. I did a hard restart on cloudera-scm-agent but I still have the same issue.

Another thing, might be related - all my services are giving this configuration warning - "Mismatched CDH versions: host has NONE but role expects 5". Recently, one of the configuration items on cloudera management service changed from "hostname" to "hostname.domainname.com" - would this be casuing any issues? This is also a pending configuration change on a lot of services (along with setting parcel key to empty value).

[#4]

Any more suggestions? 

[#5 · Accepted solution]

I figured out what the problem was, I had placed some pig libraries in the parcels folder, so when I restrated CM service it couldn't load these libraries and was giving an exception.

[#6]

 Hi all, afte rrestarting all nodes vm (I shutdown the cluster before), I got the same problem.

For moment I started again cloudera management service, and for all services I got this error 

mismatched cdh versions host has none but role expects 5

I'm really scared since I don't have any idea why it comes, do you know what I can check or do to resolve it? any help would be reatly appreciated! THANKS a lot cmmunity!