Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile

avatar
Explorer

I'm using Impala 2.5.0 with CDH 5.7.1 parcels on RHEL 6.6/6.4 versions for my clusters and very often i could see " ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile" in cloudera scm agent logs and i've been receiving couple of alerts due to this. Could anyone help me to understand the reason for these logs.

 

I would like to know the impact of these alerts for the ongoing queries in the cluster if any?

 

Please find the logs from cloudera scm agent of the cluster.

 

-------------------------------------------------- 030---------------
[08/Aug/2016 05:42:05 +0000] 12396 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.07 max:0.08 LIFE_MAX:0.15
[08/Aug/2016 05:45:25 +0000] 12396 ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile at 'http://sample030.enterprisenet.org:25000/query_profile_encoded'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/impalad/query_monitor.py", line 527, in get_executing_query_profile
password=password)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/url_util.py", line 66, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>
[08/Aug/2016 05:45:30 +0000] 12396 ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query ids at 'http://sample030.enterprisenet.org:25000/inflight_query_ids'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/impalad/query_monitor.py", line 498, in get_executing_query_ids
password=password)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/url_util.py", line 66, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>


-------------------------------------------------------------------------26----
[08/Aug/2016 05:41:54 +0000] 9923 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.07 max:0.08 LIFE_MAX:0.18
[08/Aug/2016 05:45:22 +0000] 9923 ImpalaDaemonQueryMonitoring throttling_logger ERROR (1 skipped) Error fetching executing query ids at 'http://sample026.enterprisenet.org:25000/inflight_query_ids'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/impalad/query_monitor.py", line 498, in get_executing_query_ids
password=password)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/url_util.py", line 66, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>


----------------------------------------33-----------
quested version 6.
[08/Aug/2016 05:44:18 +0000] 1564 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[08/Aug/2016 05:45:07 +0000] 1564 ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query profile at 'http://sample033.enterprisenet.org:25000/query_profile_encoded'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/impalad/query_monitor.py", line 527, in get_executing_query_profile
password=password)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/url_util.py", line 66, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>


--------------------------------------------------------20--------------
[08/Aug/2016 05:45:02 +0000] 34729 ImpalaDaemonQueryMonitoring throttling_logger ERROR Error fetching executing query ids at 'http://sample020.enterprisenet.org:25000/inflight_query_ids'
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/impalad/query_monitor.py", line 498, in get_executing_query_ids
password=password)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/url_util.py", line 66, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
URLError: <urlopen error timed out>

Who agreed with this topic