Reply
New Contributor
Posts: 3
Registered: ‎07-21-2015
Accepted Solution

Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

Hello @all,

 

we are running CDH 5.4.4 and Cloudera Manager 5.4.3 on Centos 6.4 Systems

Everything runs fine except the impala Monitoring.

The service itself is also running fine. It seems that only the monitoring of the impala daemons on all hosts and the catalogserver is not working:

 

 

The error-Messages are as follows:

 

"The Cloudera Manager Agent got an unexpected response from this role's web server. "

 

/var/log/cloudera-scm-agent/cloudera-scm-agent.log:

[21/Jul/2015 09:33:34 +0000] 2502 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('ImpaladMonitor for impala-IMPALAD-4462e41b224353ea47cbbf50630dc528',)
[21/Jul/2015 09:51:19 +0000] 2502 ImpalaDaemonQueryMonitoring throttling_logger ERROR (3429 skipped) Error fetching metrics at 'http://cloudera01-vm.spiegel.de:25000/jsonmetrics?json'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 394, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
JSONDecodeError: Expecting property name: line 75 column 19 (char 2309)
[21/Jul/2015 09:59:12 +0000] 2502 CP Server Thread-10 _cplogging INFO 172.25.12.99 - - [21/Jul/2015:09:59:12] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Process with same id has changed: 1285-impala-IMPALAD.
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Deactivating process 1285-impala-IMPALAD
[21/Jul/2015 09:59:13 +0000] 2502 MainThread __init__ INFO Stopping query monitoring thread.
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Triggering supervisord update.
[21/Jul/2015 09:59:15 +0000] 2502 CP Server Thread-11 _cplogging INFO 172.25.12.99 - - [21/Jul/2015:09:59:15] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[21/Jul/2015 09:59:15 +0000] 2502 MainThread agent INFO Deleting process 1285-impala-IMPALAD

 

Does anybody know how to get rid of this error?

Best Regards

 

Markus

 

New Contributor
Posts: 3
Registered: ‎07-21-2015

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

forgot to say, that this thread describes nearly the same problem, but the solution did not work for us:

http://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-4-Impala-webinterface-monitorin...

 

 

 

Cloudera Employee
Posts: 578
Registered: ‎01-20-2014

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

This might be https://issues.cloudera.org/browse/IMPALA-2014

Please try this workaround and let us know if it helps
1. Log into Cloudera Manager and click on the Impala service
2. Click on Configuration
3. Under the Impala (Service Wide) category set the property named Impala Service Environment Advanced Configuration Snippet (Safety Valve) to the following value:
LC_ALL="C"
4. Click Save Changes
5. Restart Impala
Regards,
Gautam Gopalakrishnan
New Contributor
Posts: 3
Registered: ‎07-21-2015

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

Hello Gautam,

 

thanks for the answer. And sorry about the delay.

 

I changed the property and restarted impala service, but the error occured again:

 

[22/Jul/2015 12:51:27 +0000] 2502 ImpalaDaemonQueryMonitoring throttling_logger ERROR (3318 skipped) Error fetching metrics at 'http://cloudera01-vm.spiegel.de:25000/jsonmetrics?json'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 394, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
JSONDecodeError: Expecting property name: line 75 column 19 (char 2309)

 

 

But - then i changed the locale from de_DE.UTF-8 to en_US.UTF-8 systemwide (in /etc/sysconfig/i18n) and the impala monitoring works fine!

 

Thanks for the help

 

Markus

Cloudera Employee
Posts: 578
Registered: ‎01-20-2014

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

Thanks for the update.
Regards,
Gautam Gopalakrishnan
Explorer
Posts: 6
Registered: ‎12-28-2017

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

[ Edited ]

Thanks,

 

I have been facing the same issue and this helped me to resolve it.

 

I would really appreciate if you explain me what exactly does this option do ? 

 

 

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

@skhadrao,

 

Generally, Impala does not support localization at this point so we have not coded to handle a difference in "decimal" points might be "commas" in some localizations.

 

The workaround is to make sure that impala roles are started with a locale that will ensure that JMX output (parsed by the Cloudera Manager agents) returns expected output.

 

LC_ALL is an environment variable for your shell/OS.  It forces a particular Locale to be used.

You can type locale and hit enter on your host to see your current locale environment variables.

 

When you specify LC_ALL in the Impala Safety Valve, this will make sure that your OS configuration remains the same but Impala roles will use the specified locale.  The end result is that the JMX information returned by Impala when the Cloudera Manager agent attempts to parse the JSON response will be compatible.

 

Ben

New Contributor
Posts: 1
Registered: ‎04-12-2018

Re: Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

It's work for me!
Thanks.
Announcements