Support Questions
Find answers, ask questions, and share your expertise

Cloudera management service and all services timeout

Solved Go to solution

Cloudera management service and all services timeout

Explorer

Hello,

 

My cluster was working fine and I hadn't monitored it for a few days. Recently when I saw, the services were not displaying their health. I tried restarting the cloudera management service and it gave me the following error -

"Command timed out after 150 seconds".

 

When I tried restarting any service, I get the same error message - and none of the services are working. Where can I start debugging the issue? Where are the configuration files of cloudera management service stored? 

 

cloudera-scm-agent, cloudera-scm-server, cloudera-scm-server-db are all running

 

 

 

 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cloudera management service and all services timeout

Explorer

I figured out what the problem was, I had placed some pig libraries in the parcels folder, so when I restrated CM service it couldn't load these libraries and was giving an exception.

View solution in original post

5 REPLIES 5

Re: Cloudera management service and all services timeout

Are the nodes able to contact each other? Try telnet to port 9000 (CM Agent's port) from one host to another. Ping nodes from each other, ensure all of them can talk to the other. If the agents are running, you should be able to connect via telnet. The easiest action to take right now is to restart the agents on the nodes. Perform this on just one or a few nodes and try restarting services on them.

 

# service cloudera-scm-agent hard_restart  (this will restart any running services as well)

 

Regards,
Gautam Gopalakrishnan

Re: Cloudera management service and all services timeout

Explorer

Thanks Gautam for the quick reply.

 

I am able to telnet on port 9000. I did a hard restart on cloudera-scm-agent but I still have the same issue.

 

Another thing, might be related - all my services are giving this configuration warning - "Mismatched CDH versions: host has NONE but role expects 5". Recently, one of the configuration items on cloudera management service changed from "hostname" to "hostname.domainname.com" - would this be casuing any issues? This is also a pending configuration change on a lot of services (along with setting parcel key to empty value).

Re: Cloudera management service and all services timeout

Explorer

Any more suggestions? 

Re: Cloudera management service and all services timeout

Explorer

I figured out what the problem was, I had placed some pig libraries in the parcels folder, so when I restrated CM service it couldn't load these libraries and was giving an exception.

View solution in original post

Re: Cloudera management service and all services timeout

Explorer

 Hi all, afte rrestarting all nodes vm (I shutdown the cluster before), I got the same problem.

 

For moment I started again cloudera management service, and for all services I got this error 

mismatched cdh versions host has none but role expects 5

 

I'm really scared since I don't have any idea why it comes, do you know what I can check or do to resolve it? any help would be reatly appreciated! THANKS a lot cmmunity!