04-27-2017 09:50 AM
Hi ,
I have upgraded CDH from 5.9 to 5.11 mainloy due to Impala statistics issue .
After the upgarde completes I started to recieve Query Monitoring status check failures .
I saw that some of the queries are also not presented in the queries pages in Impala service .
I checked the agent logs on the nodes with status check failures and these is the errrors I see :
*******************************************************************************************
[27/Apr/2017 15:52:53 +0000] 29083 ImpalaDaemonQueryMonitoring throttling_logger ERROR (358 skipped) Error fetching executing query ids at 'http://gc-dp-pdpprd-data-20.c.bi-environment-1271.internal:25000/inflight_query_ids'
Traceback (most recent call last):
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 497, in get_executing_query_ids
password=password)
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/url_util.py", line 67, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
****************************************************************************
Do you know what can be the problem ? how to solve it ?
Also statistics gathering on partition with 54 million rows takes 50 minutes .
All Other services health check seems to work good .
Thanks
Alon
04-28-2017 04:22 PM
Alon,
I'm sorry that this is not answer to your question.
actually, I have question for you.
In order to use the python package you're using, do I have to run cloudera manager on my cluster?
Thank you
Gatsby
05-01-2017 01:54 PM
Hi Alon,
CM is requesting query status from Impala's webserver, but the request is timing out. This can happen if the Impala webserver is under load or if there are network issues. How frequently is this error happening?
Thanks,
Joe
06-12-2017 03:43 AM
Hi Joe , sorry for the late reply .
It still happen - frequency is low .
The machines are in google compute engines so dont believe its network issues and the and the cluster is not so busy .
It did not happen in 5.9 .
is there a way to change configuration of Impala's webserver so it will be able to handle these requests .
Thanks
Alon