Could you please be more specific with "issue with your network"?
Also, which resources would you recommend to look for? Attached you can see memory and cpu utilization as presented by Cloudera host monitor. To me they don't look particulary impressive? (the timestamp refers to the last time this issue occurred)
Network-wise the cluster is fully isolated from the external world, and I don't find unknown ips.
we are experiencing the same or similar problem. We get a lot of (in cloudera-scm-agent.log):
[10/Aug/2017 08:00:33 +0000] 11211 ImpalaDaemonQueryMonitoring throttling_logger ERROR (31 skipped) Error fetching executing query profile at 'http://our_host_name:25000/query_profile_encoded'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 526, in get_executing_query_profile
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.1-py2.7.egg/cmf/url_util.py", line 67, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
This results in a IMPALAD_QUERY_MONITORING_STATUS alert. We are running CDH 5.10.1 on Ubuntu 14. I guess the load is fairly high on the nodes but not through the roof.