After CDH upgrade from 5.9 to 5.11 Impala query monitoring is not working properly

[AlonEldi]

Hi ,

I have upgraded CDH from 5.9 to 5.11 mainloy due to Impala statistics issue .

After the upgarde completes I started to recieve Query Monitoring status check failures .

I saw that some of the queries are also not presented in the queries pages in Impala service .

I checked the agent logs on the nodes with status check failures and these is the errrors I see :

*******************************************************************************************

[27/Apr/2017 15:52:53 +0000] 29083 ImpalaDaemonQueryMonitoring throttling_logger ERROR    (358 skipped) Error fetching executing query ids at 'http://gc-dp-pdpprd-data-20.c.bi-environment-1271.internal:25000/inflight_query_ids'
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 497, in get_executing_query_ids
    password=password)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/url_util.py", line 67, in urlopen_with_timeout
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
timeout: timed out

****************************************************************************

Do you know what can be the problem  ? how to solve it ?

Also statistics gathering on partition with 54 million rows takes 50 minutes .

All Other services health check seems to work good .

Thanks

Alon

[Gatsby]

Alon,

I'm sorry that this is not answer to your question.

actually, I have question for you.

In order to use the python package you're using, do I have to run cloudera manager on my cluster?

Thank you

Gatsby

[joemcdonnell]

Hi Alon,

CM is requesting query status from Impala's webserver, but the request is timing out. This can happen if the Impala webserver is under load or if there are network issues. How frequently is this error happening?

Thanks,

Joe

[AlonEldi]

Hi Joe , sorry for the late reply .

It still happen - frequency is low .

The machines are in google compute engines so dont believe its network issues and the and the cluster is not so busy .

It did not happen in 5.9 .

is there a way to change configuration of Impala's webserver so it will be able to handle these requests .

Thanks

Alon