Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Management services don't work after upgrade to 5.10

Highlighted

Management services don't work after upgrade to 5.10

New Contributor

After upgrading manager to 5.10 and all agents to 5.10, all services and hosts appear as "Unknown Health" and nothing work in the manager.

 

On agents I have errors like : 

[10/Feb/2017 19:27:31 +0000] 64743 MonitorDaemon-Reporter throttling_logger ERROR (12 skipped) Error sending messages to firehose: mgmt1-SERVICEMONITOR-4c5c24980753678ebe83e319f270d1e4
Traceback (most recent call last):
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
self._port)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib/python2.7/httplib.py", line 757, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused

 

 

6 REPLIES 6
Highlighted

Re: Management services don't work after upgrade to 5.10

Champion
Lets start with the agents. From a host running the agent are you able to connect to the CM server over port 7182.

Double check that hostname resolution is operating correctly?

python -c "import socket; print socket.getfqdn(); print
socket.gethostbyname(socket.getfqdn())"

Does this return the proper FQDN of the hosts?

Is there anything in the CM logs?
Highlighted

Re: Management services don't work after upgrade to 5.10

New Contributor

yes, and yes.

 

The only error in cloudera-scm-server.log are errors about parcels distribution, but nothing really worrying.

 

After trying a lot of things, I noticed that the services, not only management services, are in fact running, but they stay in 'STARTING' mode, which prevent the CM to do anything. For exemple I can start hdfs, CM will report that started fails, but I can connect the namenode webui.

 

I have this error when starting any service :

 

Screen Shot 2017-02-11 at 23.24.22.png

 

 

 

 

 

 

 

 

 

 

 

 "org.hibernate.PropertyAccessException: Exception occurred inside setter of com.cloudera.cmf.model.DbProcess.resourcesForDb"

Highlighted

Re: Management services don't work after upgrade to 5.10

Champion
Shutdown all of the services, shutdown the agents, restart Cloudera Manager server. Then start the agents back up. Are they able to connect after this or are they still getting the connection refused error?
Highlighted

Re: Management services don't work after upgrade to 5.10

New Contributor

Did all that, multiple times. Still the same.

 

Agent could always communicate with CM, I have a heartbeat for all of them.

 

But starting of services still fails, I have this error each time :

 

Screen Shot 2017-02-12 at 10.04.45.png

 

and not a single warning or error in the agent and CM logs.

 

Highlighted

Re: Management services don't work after upgrade to 5.10

Champion
Highlighted

Re: Management services don't work after upgrade to 5.10

Champion
I found a similar (but not the same exception) in another post on here. Check it out. The commands are specific to the postgres DB used if you used the installer.

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Up-and-downgrade-failure-5-4-5-lt-gt...
Don't have an account?
Coming from Hortonworks? Activate your account here