Impala monitor: error fetching completed queries

JoaquinS · ‎04-19-2017

Hello,

I'm getting this error very often from the Cloudera Monitor:

[19/Apr/2017 12:56:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 902, in _get_completed_query_profiles
    self._query_monitor.get_completed_queries(query_log_file)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 582, in get_completed_queries
    completed_query_report_limit)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 457, in get_completed_queries
    last_accessed_file_timestamp)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 257, in _get_completed_queries
    file_filter=filters)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 114, in __init__
    self.__filtered_file_list = self.__apply_file_filter()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 186, in __apply_file_filter
    self.__file_filter(filter_context)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsCommon-0.1-py2.7.egg/clusterstats/common/chain.py", line 22, in __call__
    succ = command(context)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 96, in __call__
    self.__set_end_offset(f, evt2.get_datetime())
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 116, in __set_end_offset
    f.set_end_offset(event.get_offset())
AttributeError: 'NoneType' object has no attribute 'get_offset'

This is the frecuency that i'm getting this error:

[16/Apr/2017 16:52:04 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[16/Apr/2017 20:44:04 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[18/Apr/2017 08:42:05 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[18/Apr/2017 15:08:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[18/Apr/2017 22:22:05 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 00:10:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 00:46:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 01:02:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 02:02:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 02:34:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 03:22:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 03:42:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 04:02:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 04:16:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 05:38:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 06:32:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 07:22:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 07:48:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 08:04:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 10:14:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 10:20:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 11:24:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 12:56:07 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'
[19/Apr/2017 13:06:06 +0000] 118840 ImpalaDaemonQueryMonitoring query_monitor ERROR    Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log'

I'm using CDH 5.8.0-1.cdh5.8.0.p0.42 and IMPALA_KUDU 2.7.0-1.cdh5.9.0.p0.11

Regards,

Joaquín Silva

datagrinder · ‎05-22-2018

I'm experiencing the same problem, any workaround here?