Weirdest reason for Cloudera Management Services not to start - missing parcels_cache or repo

[#1]

In our testing environments we're aleays short on space...

So we thought that we can remove the /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo - after all the parcel is already installed and distributed.

Big mistake...

i failed to start the cloudera manager services and got error like:

=========

[29/Jun/2015 10:57:41 +0000] 2917 MainThread util INFO Using generic parser for process cloudera-mgmt-HOSTMONITOR
[29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Activating Process 1019-cloudera-mgmt-HOSTMONITOR
[29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR
[29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to cloudera-scm (781) cloudera-scm (501)
[29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to 0751
[29/Jun/2015 10:57:41 +0000] 2917 MainThread agent ERROR Failed to activate {u'refresh_files': [u'cloudera-stack-monitor.properties'], u'config_generation': 0, u'auto_restart': True, u'running': True, u'required_tags': [], u'user': u'cloudera-scm', u'special_file_info': [], u'group': u'cloudera-scm', u'id': 1019, u'status_links': {u'status': u'http://ilmtxbda146.corp.amdocs.com:8091/'}, u'name': u'cloudera-mgmt-HOSTMONITOR', u'run_generation': 1, u'environment': {u'MGMT_LOG_FILE': u'mgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out', u'CDH_VERSION': u'-1', u'FIREHOSE_JAVA_OPTS': u'-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError={{AGENT_COMMON_DIR}}/killparent.sh'}, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'firehose', u'--pipeline-type', u'HOST_MONITORING'], u'parcels': {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}, u'resources': [{u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': {u'limit_memlock': None, u'limit_fds': None}, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 8091}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9995}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9994}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/lib/cloudera-host-monitor', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}, u'cpu': None}], u'one_off': False}
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1234, in handle_heartbeat_response
new_process.activate()
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2438, in activate
self.update_process_environment_and_parcels()
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2601, in update_process_environment_and_parcels
parcels_in_use = self.agent.repo.prepare_environment(self.raw['parcels'],
AttributeError: 'NoneType' object has no attribute 'prepare_environment'
[29/Jun/2015 10:57:57 +0000] 2917 CP Server Thread-12 _cplogging INFO 10.233.164.119 - - [29/Jun/2015:10:57:57] "GET /download_log?path=%2Fvar%2Flog%2Fcloudera-scm-firehose%2Fmgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out&onlyTail=true&compress=false HTTP/1.1" 200 2011 "" "Java/1.7.0_10"
[29/Jun/2015 11:07:28 +0000] 2917 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-14f3c3b6446d19e4dcc4dab16355bb71
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 70, in _send
self._port)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 464, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused

=========

Luckily for me, I had the same parcel on a different environment so I could copy it and environment was happy again.

My questions for cloudera people:

* Why do I need the parcels after it was successfully installed? (all the jars and exe are available from /opt/cloudera/parcels/CDH)

* Do you have any suggestions for saving space? - again this is for small testing environments and not for production.

Thanks

Doron

[#2 · Accepted solution]

Actually, by design, its meant to operate that way.  All cluster nodes must be at the same release level, and this is verified on startup.  What you did was pull the resources out from under the database and trigger a server service halt because it could not audit deployed vs staged parcels.  If we threw away the CDH parcel, how would you add servers and additional cluster service role instances?  Cloudera Manager can manage multiple CDH clusters that may at be varying release levels too. 

We go through the disk requirements here: 

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_reqs_space.html

The cluster maintains parcel deployment based on the UI within the CM server, you can not manage these from the command line (by doing something like removing the files from the path).  If there are additional/old unused parces you can remove them using the parcel management screen.

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_parcels.html

You can get the CM server down to where its only storing the current parcel in its repo, using the UI. Having Cloudera Manager on a CDH cluster node will drive your storage issues as well.

[#3]

Hi
Thanks for the quick replay.
I knew that the fault was mine, but it took me a while to find it.
What I would like to be able to do is to have the parcels located on a shared storage, so I cam have a single copy of the parcel file (1.5GB per parcel) for my various clusters.
In any case i have my testing clusters installed on VMs, so it does make sense to do this for them.
For real Production environment i can't argue with your solution.

Cheers
Doron

[#4]

I have the same issue using 5.5 quickstart VM. Unfortunately downloading and activating the parcels do not resolve the issue.

Every time when VM is started, the status becomes unknow, and it is fine after manually restart the Cloudera Management Service. The error is 'com.cloudera.cmf.BasicScmProxy connection refused'. It is such a bad experience of starting using Cloudera.