Support Questions
Find answers, ask questions, and share your expertise

CDH 5.4 Impala webinterface monitoring

Contributor

Hi,

 

 

I installed CDH 5.4 and now the manager interface is alerting that the Impala web interface for the catalog server, statestore and daemons are not reachable. They are however up and running. And Impala is also working fine.

 

I have configured the websites to use TLS and this worked find with CDH 5.3

 

right now i have disabled monitoring for the impala web interfaces so the service does not shows up as having trouble.

 

Any ideas on how to fix this?

1 ACCEPTED SOLUTION

Cloudera Employee

Good to know.

 

Yes, I believe this can happen if the agents have not been upgraded to CM 5.4. Impala changed the URL from which to fetch metrics in CDH 5.4 the correspondin change to the CM agent is in CM 5.4.

 

Thanks for your help nailing this down.

 

chris

View solution in original post

14 REPLIES 14

New Contributor

Our cluster has the same problem, and we faced this issue 3 times and still don't have solution.

 

Any help welocome.

Cloudera Employee

Could you post the exact text or a screenshot of the alert you're seeing? If you could post or email me the CM agent.log (/var/log/cloudera-scm-agent.log) that might help narrow this down as well.

 

We did make a change in this area in CM 5.4, but it was to make monitoring of the TLS enabled Catalog Server web UI work properly.

 

Contributor

I am running the catalog server and the statestore without SSL but the impala daemon ui is configured with ssl.

and i am getting and error for all of them, they work fine however.

 

CM has the following error msg:

 

  • The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server.

 

i get this from the log:

 

[08/May/2015 00:55:17 +0000] 21170 Monitor-CatalogServerMonitor throttling_logger ERROR (59 skipped) Error fetching metrics at 'http://xxxx:25020/jsonmetrics'
Traceback (most recent call last):
File "/usr/lib/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 409, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/simplejson-2.1.2-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 420, in raw_decode
raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

 

New Contributor

Exact the same errors in agent log.

Cloudera Employee

Thnaks. That's helpful. I've got a few more questions:

 

#1. What version of Impala are you running? If you hit 'XXX/jsonmetrics' on the catalog server web UI what do you see? What about 'XXX/jsonmetrics?json'.

 

#2. Can you confirm that the CM agent process running is 5.4? That is: when you upgraded to 5.4 did you both upgrade the CM agent software and restart the CM agent software (service cloudera-scm-agent restart).

 

#3. Can you post the contents of the cloudera-monitor.properties for the catalog server? You can get this by going to the catalog server status page -> processes tab -> configuration files -> cloudera-monitor.properties.

Contributor

 

 

The problem is fixed for me now, i upgraded CM (and all the agents) to version 5.4 from CDH-5.3.0-1.cdh5.3.0.p0.30 and now the errors are gone.

so CM 5.3.0 with CH 5.4 might cause this issue. 

Cloudera Employee

Good to know.

 

Yes, I believe this can happen if the agents have not been upgraded to CM 5.4. Impala changed the URL from which to fetch metrics in CDH 5.4 the correspondin change to the CM agent is in CM 5.4.

 

Thanks for your help nailing this down.

 

chris

New Contributor

Hello,

 

Our parcells and CM has the same version, our cloudera manager agents are running on the same version.

From agent logs while grepping CDH5.4

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel       INFO     Activating system symlinks for parcel CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:11:49 +0000] 5792 MainThread parcel       INFO     Ensuring alternatives entries are activated for parcel CDH-5.4.0-1.cdh5.4.0.p0.27.

[07/May/2015 14:11:52 +0000] 5792 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.4.0-1.cdh5.4.0.p0.27

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:12:13 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hbase-plugin']

[07/May/2015 14:12:34 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'yarn-plugin', u'mapreduce2-plugin']

[07/May/2015 14:13:09 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hive-plugin']

[07/May/2015 14:13:55 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:34 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:15:57 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:16:50 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.4.0-1.cdh5.4.0.p0.27'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']

[07/May/2015 14:17:48 +0000] 5792 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.0-1.cdh5.4.0.p0.27'}

 

From

xxx/jsonmetrics?json

{
    "__common__": {
        "process-name": "catalogd",
        "navbar": [
            {
                "link": "/",
                "title": "/"
            },
            {
                "link": "/catalog",
                "title": "/catalog"
            },
            {
                "link": "/logs",
                "title": "/logs"
            },
            {
                "link": "/memz",
                "title": "/memz"
            },
            {
                "link": "/metrics",
                "title": "/metrics"
            },
            {
                "link": "/rpcz",
                "title": "/rpcz"
            },
            {
                "link": "/threadz",
                "title": "/threadz"
            },
            {
                "link": "/varz",
                "title": "/varz"
            }
        ]
    },
    "catalog-server.topic-processing-time-s": {
        "count": 206739,
        "last": 1,4435e-05,
        "min": 4,249e-06,
        "max": 1,63532,
        "mean": 3,04206e-05,
        "stddev": 0,00433999
    },
    "catalog.version": "catalogd version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)",
    "impala.thrift-server.CatalogService.connections-in-use": 2,
    "impala.thrift-server.CatalogService.total-connections": 2,
    "rpc-method.catalog-server.CatalogService.ExecDdl.call_duration": {
        "count": 4,
        "last": 1,
        "min": 1,
        "max": 256,
        "mean": 88,5,
        "stddev": 103,968
    },
    "statestore-subscriber.heartbeat-interval-time": {
        "count": 413455,
        "last": 1,00029,
        "min": 0,716989,
        "max": 1,0054,
        "mean": 1,00019,
        "stddev": 0,000569971
    },
    "statestore-subscriber.registration-id": "ca4d49dc0341989c:43aa2ad587f4f291",
    "statestore-subscriber.statestore.client-cache.clients-in-use": 0,
    "statestore-subscriber.statestore.client-cache.total-clients": 1,
    "statestore-subscriber.topic-update-duration": {
        "count": 206738,
        "last": 8,574e-06,
        "min": 3,898e-06,
        "max": 0,000617106,
        "mean": 1,01547e-05,
        "stddev": 4,97389e-06
    },
    "statestore-subscriber.topic-update-interval-time": {
        "count": 206738,
        "last": 2,00068,
        "min": 1,7276,
        "max": 9,45017,
        "mean": 2,00027,
        "stddev": 0,0166997
    },
    "tcmalloc.bytes-in-use": 224907312,
    "tcmalloc.pageheap-free-bytes": 106496,
    "tcmalloc.pageheap-unmapped-bytes": 210419712,
    "tcmalloc.physical-bytes-reserved": 264019968,
    "tcmalloc.total-bytes-reserved": 474439680,
    "thread-manager.running-threads": 8,
    "thread-manager.total-threads-created": 8,
    "rpc-method.statestore-subscriber.StatestoreSubscriber.Heartbeat.call_duration": {
        "count": 413455,
        "last": 0,
        "min": 0,
        "max": 2,
        "mean": 0,0326976,
        "stddev": 0,177858
    },
    "rpc-method.statestore-subscriber.StatestoreSubscriber.UpdateState.call_duration": {
        "count": 206738,
        "last": 0,
        "min": 0,
        "max": 108,
        "mean": 0,0556792,
        "stddev": 0,379281
    },
    "statestore-subscriber.connected": true,
    "statestore-subscriber.last-recovery-duration": 0,
    "statestore-subscriber.last-recovery-time": "N/A",
    "statestore-subscriber.topic-catalog-update.processing-time-s": {
        "count": 206738,
        "last": 7,327e-06,
        "min": 3,249e-06,
        "max": 0,000614711,
        "mean": 8,50139e-06,
        "stddev": 4,79989e-06
    },
    "jvm.code-cache.committed-usage-bytes": 5505024,
    "jvm.code-cache.current-usage-bytes": 5389824,
    "jvm.code-cache.init-usage-bytes": 2555904,
    "jvm.code-cache.max-usage-bytes": 50331648,
    "jvm.code-cache.peak-committed-usage-bytes": 5505024,
    "jvm.code-cache.peak-current-usage-bytes": 5396544,
    "jvm.code-cache.peak-init-usage-bytes": 2555904,
    "jvm.code-cache.peak-max-usage-bytes": 50331648,
    "jvm.ps-eden-space.committed-usage-bytes": 263716864,
    "jvm.ps-eden-space.current-usage-bytes": 31283240,
    "jvm.ps-eden-space.init-usage-bytes": 537919488,
    "jvm.ps-eden-space.max-usage-bytes": 10705436672,
    "jvm.ps-eden-space.peak-committed-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-current-usage-bytes": 1748500480,
    "jvm.ps-eden-space.peak-init-usage-bytes": 537919488,
    "jvm.ps-eden-space.peak-max-usage-bytes": 10706485248,
    "jvm.ps-old-gen.committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.max-usage-bytes": 21417164800,
    "jvm.ps-old-gen.peak-committed-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-current-usage-bytes": 1049145432,
    "jvm.ps-old-gen.peak-init-usage-bytes": 1431306240,
    "jvm.ps-old-gen.peak-max-usage-bytes": 21417164800,
    "jvm.ps-perm-gen.committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.max-usage-bytes": 85983232,
    "jvm.ps-perm-gen.peak-committed-usage-bytes": 31457280,
    "jvm.ps-perm-gen.peak-current-usage-bytes": 31360176,
    "jvm.ps-perm-gen.peak-init-usage-bytes": 22020096,
    "jvm.ps-perm-gen.peak-max-usage-bytes": 85983232,
    "jvm.ps-survivor-space.committed-usage-bytes": 2097152,
    "jvm.ps-survivor-space.current-usage-bytes": 575480,
    "jvm.ps-survivor-space.init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.max-usage-bytes": 2097152,
    "jvm.ps-survivor-space.peak-committed-usage-bytes": 345505792,
    "jvm.ps-survivor-space.peak-current-usage-bytes": 274171664,
    "jvm.ps-survivor-space.peak-init-usage-bytes": 89128960,
    "jvm.ps-survivor-space.peak-max-usage-bytes": 345505792,
    "jvm.total.committed-usage-bytes": 1734082560,
    "jvm.total.current-usage-bytes": 1117754152,
    "jvm.total.init-usage-bytes": 2082930688,
    "jvm.total.max-usage-bytes": 32261013504,
    "jvm.total.peak-committed-usage-bytes": 3562274816,
    "jvm.total.peak-current-usage-bytes": 3108574296,
    "jvm.total.peak-init-usage-bytes": 2082930688,
    "jvm.total.peak-max-usage-bytes": 32605470720
}

New Contributor

xxx/jsonmetrics

 

Could not open template: /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/impala//www//legacy-metrics.tmpl

Explorer

Hey I'm still facing this issue !
I have CDH 5.4.2 and CM 5.4.1

I verified and the agents are all updated.

 

What else can I do ?

New Contributor

Hi, 

 

we have the same Problem too after upgrading to CDH 5.4.4 and CM 5.4.

the problem is the decimal delimiter 

"max": 0,259594,

 

before upgrading it was a point as delimiter 

\"max\": 1.52533,

that results in a "JSONDecodeError: Expecting property name: line 90 column 19 (char 3162)"

 

how can we fix it?

Cloudera Employee

Hello!

 

I'd guess that you are not in the US, and that you have locale (e.g., LC_ALL) environment variables set.  You're running into https://issues.cloudera.org/browse/IMPALA-2014.  In the meantime, as a workaround, place "LC_ALL=C" into the configuration "Impala Service Environment Advanced Configuration Snippet (Safety Valve)".

 

Please do report back if that works as a workaround for you!

 

Thanks!

 

-- Philip

New Contributor

Hi Philip, 

 

thanks, the workaround works.

Explorer

Thanks Philip, this worked for me too !

; ;