Cloudera-scm-agent metrics collection exception [ Edited ]

[sbpothineni]

Metrics collection is often pausing in cloudera manager 5.14.1, we see following exception in the cloudera-scm-agent log, replaced hostname with XXX for privacy

Error fetching metrics at 'http://XXXXXXXXXXXXXX:8042/jmx'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.1-py2.7.egg/cmf/monitor/generic/metric_collectors.py", line 203, in _collect_and_parse_and_return
simplejson.load(opened_url))
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
JSONDecodeError: Expecting object: line 388 column 6 (char 98621)

Any thoughts?

[rufusayeni]

Hello sbpothineni,

Based on the limited log snippet, it appears you may need to increase the heap size for your NodeManager.

You can increase the NodeManager's heap size by doing the following:

1. In CM UI, go to YARN -> Configuration -> NodeManager -> Java Heap Size of NodeManager in Bytes.

2. Change the value.

3. Save changes and redeploy configuration.