thanks for feedback.
Could you please be more specific with "issue with your network"?
Also, which resources would you recommend to look for? Attached you can see memory and cpu utilization as presented by Cloudera host monitor. To me they don't look particulary impressive? (the timestamp refers to the last time this issue occurred)
Network-wise the cluster is fully isolated from the external world, and I don't find unknown ips.
Thanks and regards,
we are experiencing the same or similar problem. We get a lot of (in cloudera-scm-agent.log):
[10/Aug/2017 08:00:33 +0000] 11211 ImpalaDaemonQueryMonitoring throttling_logger ERROR (31 skipped) Error fetching executing query profile at 'http://our_host_name:25000/query_profile_encoded' Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 526, in get_executing_query_profile password=password) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/url_util.py", line 67, in urlopen_with_timeout return opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open r = h.getresponse(buffering=True) File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse response.begin() File "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) timeout: timed out
This results in a IMPALAD_QUERY_MONITORING_STATUS alert. We are running CDH 5.10.1 on Ubuntu 14. I guess the load is fairly high on the nodes but not through the roof.