Problems with impalad 2.2.0 and CDH 5.4.4: The health of the Impala Catalog Server is bad

[#1]

Hello @All,

we are running CDH 5.4.4 and Cloudera Manager 5.4.3 on Centos 6.4 Systems

Everything runs fine except the impala Monitoring.

The service itself is also running fine. It seems that only the monitoring of the impala daemons on all hosts and the catalogserver is not working:

impala: Health Issues

• catalogserver (cloudera06-vm)
  Web Server Status
• impalad (cloudera01-vm)
  Impala Daemon Ready Check, Web Server Status
• impalad (cloudera02-vm)
  Impala Daemon Ready Check, Web Server Status
• impalad (cloudera03-vm)
  Impala Daemon Ready Check, Web Server Status
• impalad (cloudera04-vm)
  Impala Daemon Ready Check, Web Server Status
• impalad (cloudera05-vm)
  Impala Daemon Ready Check, Web Server Status

The error-Messages are as follows:

"The Cloudera Manager Agent got an unexpected response from this role's web server. "

/var/log/cloudera-scm-agent/cloudera-scm-agent.log:

[21/Jul/2015 09:33:34 +0000] 2502 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('ImpaladMonitor for impala-IMPALAD-4462e41b224353ea47cbbf50630dc528',)
[21/Jul/2015 09:51:19 +0000] 2502 ImpalaDaemonQueryMonitoring throttling_logger ERROR (3429 skipped) Error fetching metrics at 'http://cloudera01-vm.spiegel.de:25000/jsonmetrics?json'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 394, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
JSONDecodeError: Expecting property name: line 75 column 19 (char 2309)
[21/Jul/2015 09:59:12 +0000] 2502 CP Server Thread-10 _cplogging INFO 172.25.12.99 - - [21/Jul/2015:09:59:12] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Process with same id has changed: 1285-impala-IMPALAD.
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Deactivating process 1285-impala-IMPALAD
[21/Jul/2015 09:59:13 +0000] 2502 MainThread __init__ INFO Stopping query monitoring thread.
[21/Jul/2015 09:59:13 +0000] 2502 MainThread agent INFO Triggering supervisord update.
[21/Jul/2015 09:59:15 +0000] 2502 CP Server Thread-11 _cplogging INFO 172.25.12.99 - - [21/Jul/2015:09:59:15] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[21/Jul/2015 09:59:15 +0000] 2502 MainThread agent INFO Deleting process 1285-impala-IMPALAD

Does anybody know how to get rid of this error?

Best Regards

Markus

[#2]

forgot to say, that this thread describes nearly the same problem, but the solution did not work for us:

http://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-4-Impala-webinterface-monitorin...

[#3]

This might be https://issues.cloudera.org/browse/IMPALA-2014

Please try this workaround and let us know if it helps
1. Log into Cloudera Manager and click on the Impala service
2. Click on Configuration
3. Under the Impala (Service Wide) category set the property named Impala Service Environment Advanced Configuration Snippet (Safety Valve) to the following value:
LC_ALL="C"
4. Click Save Changes
5. Restart Impala

[#4 · Accepted solution]

Hello Gautam,

thanks for the answer. And sorry about the delay.

I changed the property and restarted impala service, but the error occured again:

[22/Jul/2015 12:51:27 +0000] 2502 ImpalaDaemonQueryMonitoring throttling_logger ERROR (3318 skipped) Error fetching metrics at 'http://cloudera01-vm.spiegel.de:25000/jsonmetrics?json'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/abstract_monitor.py", line 394, in collect_metrics_from_url
result = self.parse_metrics(simplejson.load(openedUrl))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 328, in load
use_decimal=use_decimal, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 384, in loads
return _default_decoder.decode(s)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 402, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/simplejson-2.1.2-py2.6-linux-x86_64.egg/simplejson/decoder.py", line 418, in raw_decode
obj, end = self.scan_once(s, idx)
JSONDecodeError: Expecting property name: line 75 column 19 (char 2309)

But - then i changed the locale from de_DE.UTF-8 to en_US.UTF-8 systemwide (in /etc/sysconfig/i18n) and the impala monitoring works fine!

Thanks for the help

Markus

[#5]

Thanks for the update.

[#6]

Thanks,

I have been facing the same issue and this helped me to resolve it.

I would really appreciate if you explain me what exactly does this option do ? 

[#7]

@skhadrao,

Generally, Impala does not support localization at this point so we have not coded to handle a difference in "decimal" points might be "commas" in some localizations.

The workaround is to make sure that impala roles are started with a locale that will ensure that JMX output (parsed by the Cloudera Manager agents) returns expected output.

LC_ALL is an environment variable for your shell/OS.  It forces a particular Locale to be used.

You can type locale and hit enter on your host to see your current locale environment variables.

When you specify LC_ALL in the Impala Safety Valve, this will make sure that your OS configuration remains the same but Impala roles will use the specified locale.  The end result is that the JMX information returned by Impala when the Cloudera Manager agent attempts to parse the JSON response will be compatible.

Ben

[#8]

It's work for me!
Thanks.