Reply
Explorer
Posts: 18
Registered: ‎10-03-2015

Re: ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile

Hi,

thanks for feedback.

Could you please be more specific with "issue with your network"?

 

Also, which resources would you recommend to look for? Attached you can see memory and cpu utilization as presented by Cloudera host monitor. To me they don't look particulary impressive? (the timestamp refers to the last time this issue occurred)

 

memory.pngcpu.png

 

 

Network-wise the cluster is fully isolated from the external world, and I don't find unknown ips.

 

Thanks and regards,

FIlippo

Highlighted
Explorer
Posts: 20
Registered: ‎01-07-2016

Re: ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile

Hi,

 

we are experiencing the same or similar problem. We get a lot of (in cloudera-scm-agent.log):

 

[10/Aug/2017 08:00:33 +0000] 11211 ImpalaDaemonQueryMonitoring throttling_logger ERROR    (31 skipped) Error fetching executing query profile at 'http://our_host_name:25000/query_profile_encoded'
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 526, in get_executing_query_profile
    password=password)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/url_util.py", line 67, in urlopen_with_timeout
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
timeout: timed out

This results in a IMPALAD_QUERY_MONITORING_STATUS alert. We are running CDH 5.10.1 on Ubuntu 14. I guess the load is fairly high on the nodes but not through the roof.

Announcements