question Re: CDH 5.4 Impala webinterface monitoring in Support Questions

CDH 5.4 Impala webinterface monitoring

crcerror — Fri, 16 Sep 2022 09:27:31 GMT

Hi,

I installed CDH 5.4 and now the manager interface is alerting that the Impala web interface for the catalog server, statestore and daemons are not reachable. They are however up and running. And Impala is also working fine.

I have configured the websites to use TLS and this worked find with CDH 5.3

right now i have disabled monitoring for the impala web interfaces so the service does not shows up as having trouble.

Any ideas on how to fix this?

Re: CDH 5.4 Impala webinterface monitoring

dbolshak — Thu, 07 May 2015 15:45:09 GMT

Our cluster has the same problem, and we faced this issue 3 times and still don't have solution.

Any help welocome.

Re: CDH 5.4 Impala webinterface monitoring

cleroy — Fri, 08 May 2015 05:20:38 GMT

Could you post the exact text or a screenshot of the alert you're seeing? If you could post or email me the CM agent.log (/var/log/cloudera-scm-agent.log) that might help narrow this down as well.

We did make a change in this area in CM 5.4, but it was to make monitoring of the TLS enabled Catalog Server web UI work properly.

Re: CDH 5.4 Impala webinterface monitoring

crcerror — Fri, 08 May 2015 06:27:08 GMT

I am running the catalog server and the statestore without SSL but the impala daemon ui is configured with ssl.

and i am getting and error for all of them, they work fine however.

CM has the following error msg:

The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server.

i get this from the log:

[08/May/2015 00:55:17 +0000] 21170 Monitor-CatalogServerMonitor throttling_logger ERROR (59 skipped) Error fetching metrics at 'http://xxxx:25020/jsonmetrics'
Traceback (most recent call last):
File "/usr/lib/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 409, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 420, in raw_decode
raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

Re: CDH 5.4 Impala webinterface monitoring

dbolshak — Fri, 08 May 2015 06:44:23 GMT

Exact the same errors in agent log.

Re: CDH 5.4 Impala webinterface monitoring

cleroy — Fri, 08 May 2015 15:38:35 GMT

Thnaks. That's helpful. I've got a few more questions:

#1. What version of Impala are you running? If you hit 'XXX/jsonmetrics' on the catalog server web UI what do you see? What about 'XXX/jsonmetrics?json'.

#2. Can you confirm that the CM agent process running is 5.4? That is: when you upgraded to 5.4 did you both upgrade the CM agent software and restart the CM agent software (service cloudera-scm-agent restart).

#3. Can you post the contents of the cloudera-monitor.properties for the catalog server? You can get this by going to the catalog server status page -> processes tab -> configuration files -> cloudera-monitor.properties.

Re: CDH 5.4 Impala webinterface monitoring

crcerror — Fri, 08 May 2015 15:55:53 GMT

The problem is fixed for me now, i upgraded CM (and all the agents) to version 5.4 from CDH-5.3.0-1.cdh5.3.0.p0.30 and now the errors are gone.

so CM 5.3.0 with CH 5.4 might cause this issue.

Re: CDH 5.4 Impala webinterface monitoring

cleroy — Fri, 08 May 2015 15:58:46 GMT

Good to know.

Yes, I believe this can happen if the agents have not been upgraded to CM 5.4. Impala changed the URL from which to fetch metrics in CDH 5.4 the correspondin change to the CM agent is in CM 5.4.

Thanks for your help nailing this down.

chris

Re: CDH 5.4 Impala webinterface monitoring

dbolshak — Tue, 12 May 2015 06:37:56 GMT

Hello,

Our parcells and CM has the same version, our cloudera manager agents are running on the same version.

From agent logs while grepping CDH5.4

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel INFO Activating system symlinks for parcel CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel INFO Ensuring alternatives entries are activated for parcel CDH-5.4.0-1.cdh5.4.0.p0.27.

[07/May/2015 14:11:52 +0000] 5792 MainThread parcel INFO Loading parcel manifest for: CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hbase-plugin']

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'yarn-plugin', u'mapreduce2-plugin']

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin']

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel INFO prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

From

xxx/jsonmetrics?json

{
    "__common__": {
        "process-name": "catalogd",
        "navbar": [
            {
                "link": "/",
                "title": "/"
            },
            {
                "link": "/catalog",
                "title": "/catalog"
            },
            {
                "link": "/logs",
                "title": "/logs"
            },
            {
                "link": "/memz",
                "title": "/memz"
            },
            {
                "link": "/metrics",
                "title": "/metrics"
            },
            {
                "link": "/rpcz",
                "title": "/rpcz"
            },
            {
                "link": "/threadz",
                "title": "/threadz"
            },
            {
                "link": "/varz",
                "title": "/varz"
            }
        ]
    },
    "catalog-server.topic-processing-time-s": {
        "count": 206739,
        "last": 1,4435e-05,
        "min": 4,249e-06,
        "max": 1,63532,
        "mean": 3,04206e-05,
        "stddev": 0,00433999
    },
    "catalog.version": "catalogd version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)",
    "impala.thrift-server.CatalogService.connections-in-use": 2,
    "impala.thrift-server.CatalogService.total-connections": 2,
    "rpc-method.catalog-server.CatalogService.ExecDdl.call_duration": {
        "count": 4,
        "last": 1,
        "min": 1,
        "max": 256,
        "mean": 88,5,
        "stddev": 103,968
    },
    "statestore-subscriber.heartbeat-interval-time": {
        "count": 413455,
        "last": 1,00029,
        "min": 0,716989,
        "max": 1,0054,
        "mean": 1,00019,
        "stddev": 0,000569971
    },
    "statestore-subscriber.registration-id": "ca4d49dc0341989c:43aa2ad587f4f291",
    "statestore-subscriber.statestore.client-cache.clients-in-use": 0,
    "statestore-subscriber.statestore.client-cache.total-clients": 1,
    "statestore-subscriber.topic-update-duration": {
        "count": 206738,
        "last": 8,574e-06,
        "min": 3,898e-06,
        "max": 0,000617106,
        "mean": 1,01547e-05,
        "stddev": 4,97389e-06
    },
    "statestore-subscriber.topic-update-interval-time": {
        "count": 206738,
        "last": 2,00068,
        "min": 1,7276,
        "max": 9,45017,
        "mean": 2,00027,
        "stddev": 0,0166997
    },
    "tcmalloc.bytes-in-use": 224907312,
    "tcmalloc.pageheap-free-bytes": 106496,
    "tcmalloc.pageheap-unmapped-bytes": 210419712,
    "tcmalloc.physical-bytes-reserved": 264019968,
    "tcmalloc.total-bytes-reserved": 474439680,
    "thread-manager.running-threads": 8,
    "thread-manager.total-threads-created": 8,
    "rpc-method.statestore-subscriber.StatestoreSubscriber.Heartbeat.call_duration": {
        "count": 413455,
        "last": 0,
        "min": 0,
        "max": 2,
        "mean": 0,0326976,
        "stddev": 0,177858
    },
    "rpc-method.statestore-subscriber.StatestoreSubscriber.UpdateState.call_duration": {
        "count": 206738,
        "last": 0,
        "min": 0,
        "max": 108,
        "mean": 0,0556792,
        "stddev": 0,379281
    },
    "statestore-subscriber.connected": true,
    "statestore-subscriber.last-recovery-duration": 0,
    "statestore-subscriber.last-recovery-time": "N/A",
    "statestore-subscriber.topic-catalog-update.processing-time-s": {
        "count": 206738,
        "last": 7,327e-06,
        "min": 3,249e-06,
        "max": 0,000614711,
        "mean": 8,50139e-06,
        "stddev": 4,79989e-06
    },
    "jvm.code-cache.committed-usage-bytes": 5505024,
    "jvm.code-cache.current-usage-bytes": 5389824,
    "jvm.code-cache.init-usage-bytes": 2555904,
    "jvm.code-cache.max-usage-bytes": 50331648,
    "jvm.code-cache.peak-committed-usage-bytes": 5505024,
    "jvm.code-cache.peak-current-usage-bytes": 5396544,
    "jvm.code-cache.peak-init-usage-bytes": 2555904,
    "jvm.code-cache.peak-max-usage-bytes": 50331648,
    "jvm.ps-eden-space.committed-usage-bytes": 263716864,
    "jvm.ps-eden-space.current-usage-bytes": 31283240,
    "jvm.ps-eden-space.init-usage-bytes": 537919488,
    "jvm.ps-eden-space.max-usage-bytes": 10705436672,
    "jvm.ps-eden-space.peak-committed-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-current-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-init-usage-bytes": 537919488,
    "jvm.ps-eden-space.peak-max-usage-bytes": 10706485248,
    "jvm.ps-old-gen.committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.max-usage-bytes": 21417164800,
    "jvm.ps-old-gen.peak-committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.peak-init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-max-usage-bytes": 21417164800,
    "jvm.ps-perm-gen.committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.max-usage-bytes": 85983232,
    "jvm.ps-perm-gen.peak-committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.peak-current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.peak-init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.peak-max-usage-bytes": 85983232,
    "jvm.ps-survivor-space.committed-usage-bytes": 2097152,
    "jvm.ps-survivor-space.current-usage-bytes": 575480,
    "jvm.ps-survivor-space.init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.max-usage-bytes": 2097152,
    "jvm.ps-survivor-space.peak-committed-usage-bytes": 345505792,
    "jvm.ps-survivor-space.peak-current-usage-bytes": 274171664,
    "jvm.ps-survivor-space.peak-init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.peak-max-usage-bytes": 345505792,
    "jvm.total.committed-usage-bytes": 1734082560,
    "jvm.total.current-usage-bytes": 1117754152,
    "jvm.total.init-usage-bytes": 2082930688,
    "jvm.total.max-usage-bytes": 32261013504,
    "jvm.total.peak-committed-usage-bytes": 3562274816,
    "jvm.total.peak-current-usage-bytes": 3108574296,
    "jvm.total.peak-init-usage-bytes": 2082930688,
    "jvm.total.peak-max-usage-bytes": 32605470720
}

Re: CDH 5.4 Impala webinterface monitoring

dbolshak — Tue, 12 May 2015 06:40:10 GMT

xxx/jsonmetrics

Could not open template: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala//www//legacy-metrics.tmpl

Re: CDH 5.4 Impala webinterface monitoring

Omar-ITEC — Thu, 02 Jul 2015 08:00:24 GMT

Hey I'm still facing this issue !
I have CDH 5.4.2 and CM 5.4.1

I verified and the agents are all updated.

What else can I do ?

Re: CDH 5.4 Impala webinterface monitoring

elzippo3480 — Wed, 22 Jul 2015 08:40:53 GMT

Hi,

we have the same Problem too after upgrading to CDH 5.4.4 and CM 5.4.

the problem is the decimal delimiter

"max": 0,259594,

before upgrading it was a point as delimiter

\"max\": 1.52533,

that results in a "JSONDecodeError: Expecting property name: line 90 column 19 (char 3162)"

how can we fix it?

Re: CDH 5.4 Impala webinterface monitoring

philip — Wed, 22 Jul 2015 18:10:14 GMT

Hello!

I'd guess that you are not in the US, and that you have locale (e.g., LC_ALL) environment variables set. You're running into https://issues.cloudera.org/browse/IMPALA-2014. In the meantime, as a workaround, place "LC_ALL=C" into the configuration "Impala Service Environment Advanced Configuration Snippet (Safety Valve)".

Please do report back if that works as a workaround for you!

Thanks!

-- Philip

Re: CDH 5.4 Impala webinterface monitoring

elzippo3480 — Thu, 23 Jul 2015 11:02:36 GMT

Hi Philip,

thanks, the workaround works.

Re: CDH 5.4 Impala webinterface monitoring

Omar-ITEC — Fri, 07 Aug 2015 15:42:17 GMT

Thanks Philip, this worked for me too !