CDH 5.4 Impala webinterface monitoring

[#1]

Hi,

I installed CDH 5.4 and now the manager interface is alerting that the Impala web interface for the catalog server, statestore and daemons are not reachable. They are however up and running. And Impala is also working fine.

I have configured the websites to use TLS and this worked find with CDH 5.3

right now i have disabled monitoring for the impala web interfaces so the service does not shows up as having trouble.

Any ideas on how to fix this?

[#2]

Our cluster has the same problem, and we faced this issue 3 times and still don't have solution.

Any help welocome.

[#3]

Could you post the exact text or a screenshot of the alert you're seeing? If you could post or email me the CM agent.log (/var/log/cloudera-scm-agent.log) that might help narrow this down as well.

We did make a change in this area in CM 5.4, but it was to make monitoring of the TLS enabled Catalog Server web UI work properly.

[#4]

I am running the catalog server and the statestore without SSL but the impala daemon ui is configured with ssl.

and i am getting and error for all of them, they work fine however.

CM has the following error msg:

• The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server.

i get this from the log:

[08/May/2015 00:55:17 +0000] 21170 Monitor-CatalogServerMonitor throttling_logger ERROR (59 skipped) Error fetching metrics at 'http://xxxx:25020/jsonmetrics'
Traceback (most recent call last):
File "/usr/lib/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 409, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 420, in raw_decode
raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

[#5]

Exact the same errors in agent log.

[#6]

Thnaks. That's helpful. I've got a few more questions:

#1. What version of Impala are you running? If you hit 'XXX/jsonmetrics' on the catalog server web UI what do you see? What about 'XXX/jsonmetrics?json'.

#2. Can you confirm that the CM agent process running is 5.4? That is: when you upgraded to 5.4 did you both upgrade the CM agent software and restart the CM agent software (service cloudera-scm-agent restart).

#3. Can you post the contents of the cloudera-monitor.properties for the catalog server? You can get this by going to the catalog server status page -> processes tab -> configuration files -> cloudera-monitor.properties.

[#7]

The problem is fixed for me now, i upgraded CM (and all the agents) to version 5.4 from CDH-5.3.0-1.cdh5.3.0.p0.30 and now the errors are gone.

so CM 5.3.0 with CH 5.4 might cause this issue. 

[#8 · Accepted solution]

Good to know.

Yes, I believe this can happen if the agents have not been upgraded to CM 5.4. Impala changed the URL from which to fetch metrics in CDH 5.4 the correspondin change to the CM agent is in CM 5.4.

Thanks for your help nailing this down.

chris

[#9]

Hello,

Our parcells and CM has the same version, our cloudera manager agents are running on the same version.

From agent logs while grepping CDH5.4

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel       INFO     Activating system symlinks for parcel CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel       INFO     Ensuring alternatives entries are activated for parcel CDH-5.4.0-1.cdh5.4.0.p0.27.

[07/May/2015 14:11:52 +0000] 5792 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hbase-plugin']

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'yarn-plugin', u'mapreduce2-plugin']

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin']

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

From

xxx/jsonmetrics?json

{
    "__common__": {
        "process-name": "catalogd",
        "navbar": [
            {
                "link": "/",
                "title": "/"
            },
            {
                "link": "/catalog",
                "title": "/catalog"
            },
            {
                "link": "/logs",
                "title": "/logs"
            },
            {
                "link": "/memz",
                "title": "/memz"
            },
            {
                "link": "/metrics",
                "title": "/metrics"
            },
            {
                "link": "/rpcz",
                "title": "/rpcz"
            },
            {
                "link": "/threadz",
                "title": "/threadz"
            },
            {
                "link": "/varz",
                "title": "/varz"
            }
        ]
    },
    "catalog-server.topic-processing-time-s": {
        "count": 206739,
        "last": 1,4435e-05,
        "min": 4,249e-06,
        "max": 1,63532,
        "mean": 3,04206e-05,
        "stddev": 0,00433999
    },
    "catalog.version": "catalogd version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)",
    "impala.thrift-server.CatalogService.connections-in-use": 2,
    "impala.thrift-server.CatalogService.total-connections": 2,
    "rpc-method.catalog-server.CatalogService.ExecDdl.call_duration": {
        "count": 4,
        "last": 1,
        "min": 1,
        "max": 256,
        "mean": 88,5,
        "stddev": 103,968
    },
    "statestore-subscriber.heartbeat-interval-time": {
        "count": 413455,
        "last": 1,00029,
        "min": 0,716989,
        "max": 1,0054,
        "mean": 1,00019,
        "stddev": 0,000569971
    },
    "statestore-subscriber.registration-id": "ca4d49dc0341989c:43aa2ad587f4f291",
    "statestore-subscriber.statestore.client-cache.clients-in-use": 0,
    "statestore-subscriber.statestore.client-cache.total-clients": 1,
    "statestore-subscriber.topic-update-duration": {
        "count": 206738,
        "last": 8,574e-06,
        "min": 3,898e-06,
        "max": 0,000617106,
        "mean": 1,01547e-05,
        "stddev": 4,97389e-06
    },
    "statestore-subscriber.topic-update-interval-time": {
        "count": 206738,
        "last": 2,00068,
        "min": 1,7276,
        "max": 9,45017,
        "mean": 2,00027,
        "stddev": 0,0166997
    },
    "tcmalloc.bytes-in-use": 224907312,
    "tcmalloc.pageheap-free-bytes": 106496,
    "tcmalloc.pageheap-unmapped-bytes": 210419712,
    "tcmalloc.physical-bytes-reserved": 264019968,
    "tcmalloc.total-bytes-reserved": 474439680,
    "thread-manager.running-threads": 8,
    "thread-manager.total-threads-created": 8,
    "rpc-method.statestore-subscriber.StatestoreSubscriber.Heartbeat.call_duration": {
        "count": 413455,
        "last": 0,
        "min": 0,
        "max": 2,
        "mean": 0,0326976,
        "stddev": 0,177858
    },
    "rpc-method.statestore-subscriber.StatestoreSubscriber.UpdateState.call_duration": {
        "count": 206738,
        "last": 0,
        "min": 0,
        "max": 108,
        "mean": 0,0556792,
        "stddev": 0,379281
    },
    "statestore-subscriber.connected": true,
    "statestore-subscriber.last-recovery-duration": 0,
    "statestore-subscriber.last-recovery-time": "N/A",
    "statestore-subscriber.topic-catalog-update.processing-time-s": {
        "count": 206738,
        "last": 7,327e-06,
        "min": 3,249e-06,
        "max": 0,000614711,
        "mean": 8,50139e-06,
        "stddev": 4,79989e-06
    },
    "jvm.code-cache.committed-usage-bytes": 5505024,
    "jvm.code-cache.current-usage-bytes": 5389824,
    "jvm.code-cache.init-usage-bytes": 2555904,
    "jvm.code-cache.max-usage-bytes": 50331648,
    "jvm.code-cache.peak-committed-usage-bytes": 5505024,
    "jvm.code-cache.peak-current-usage-bytes": 5396544,
    "jvm.code-cache.peak-init-usage-bytes": 2555904,
    "jvm.code-cache.peak-max-usage-bytes": 50331648,
    "jvm.ps-eden-space.committed-usage-bytes": 263716864,
    "jvm.ps-eden-space.current-usage-bytes": 31283240,
    "jvm.ps-eden-space.init-usage-bytes": 537919488,
    "jvm.ps-eden-space.max-usage-bytes": 10705436672,
    "jvm.ps-eden-space.peak-committed-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-current-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-init-usage-bytes": 537919488,
    "jvm.ps-eden-space.peak-max-usage-bytes": 10706485248,
    "jvm.ps-old-gen.committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.max-usage-bytes": 21417164800,
    "jvm.ps-old-gen.peak-committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.peak-init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-max-usage-bytes": 21417164800,
    "jvm.ps-perm-gen.committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.max-usage-bytes": 85983232,
    "jvm.ps-perm-gen.peak-committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.peak-current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.peak-init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.peak-max-usage-bytes": 85983232,
    "jvm.ps-survivor-space.committed-usage-bytes": 2097152,
    "jvm.ps-survivor-space.current-usage-bytes": 575480,
    "jvm.ps-survivor-space.init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.max-usage-bytes": 2097152,
    "jvm.ps-survivor-space.peak-committed-usage-bytes": 345505792,
    "jvm.ps-survivor-space.peak-current-usage-bytes": 274171664,
    "jvm.ps-survivor-space.peak-init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.peak-max-usage-bytes": 345505792,
    "jvm.total.committed-usage-bytes": 1734082560,
    "jvm.total.current-usage-bytes": 1117754152,
    "jvm.total.init-usage-bytes": 2082930688,
    "jvm.total.max-usage-bytes": 32261013504,
    "jvm.total.peak-committed-usage-bytes": 3562274816,
    "jvm.total.peak-current-usage-bytes": 3108574296,
    "jvm.total.peak-init-usage-bytes": 2082930688,
    "jvm.total.peak-max-usage-bytes": 32605470720
}

[#10]

xxx/jsonmetrics

Could not open template: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala//www//legacy-metrics.tmpl