Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudera management service and all services timeout

avatar
Explorer

Hello,

 

My cluster was working fine and I hadn't monitored it for a few days. Recently when I saw, the services were not displaying their health. I tried restarting the cloudera management service and it gave me the following error -

"Command timed out after 150 seconds".

 

When I tried restarting any service, I get the same error message - and none of the services are working. Where can I start debugging the issue? Where are the configuration files of cloudera management service stored? 

 

cloudera-scm-agent, cloudera-scm-server, cloudera-scm-server-db are all running

 

 

 

 
1 ACCEPTED SOLUTION

avatar
Explorer

I figured out what the problem was, I had placed some pig libraries in the parcels folder, so when I restrated CM service it couldn't load these libraries and was giving an exception.

View solution in original post

5 REPLIES 5

avatar

Are the nodes able to contact each other? Try telnet to port 9000 (CM Agent's port) from one host to another. Ping nodes from each other, ensure all of them can talk to the other. If the agents are running, you should be able to connect via telnet. The easiest action to take right now is to restart the agents on the nodes. Perform this on just one or a few nodes and try restarting services on them.

 

# service cloudera-scm-agent hard_restart  (this will restart any running services as well)

 

Regards,
Gautam Gopalakrishnan

avatar
Explorer

Thanks Gautam for the quick reply.

 

I am able to telnet on port 9000. I did a hard restart on cloudera-scm-agent but I still have the same issue.

 

Another thing, might be related - all my services are giving this configuration warning - "Mismatched CDH versions: host has NONE but role expects 5". Recently, one of the configuration items on cloudera management service changed from "hostname" to "hostname.domainname.com" - would this be casuing any issues? This is also a pending configuration change on a lot of services (along with setting parcel key to empty value).

avatar
Explorer

Any more suggestions? 

avatar
Explorer

I figured out what the problem was, I had placed some pig libraries in the parcels folder, so when I restrated CM service it couldn't load these libraries and was giving an exception.

avatar
Explorer

 Hi all, afte rrestarting all nodes vm (I shutdown the cluster before), I got the same problem.

 

For moment I started again cloudera management service, and for all services I got this error 

mismatched cdh versions host has none but role expects 5

 

I'm really scared since I don't have any idea why it comes, do you know what I can check or do to resolve it? any help would be reatly appreciated! THANKS a lot cmmunity!