Thank you very much for your reply.
I check all the processes. But i didn't found something wrong.
Finally, i checked the source code of the cloudera-scm-agent(/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/agent.py), and i found this
def supervisor_check(self):
"""
Check if supervisord process is still alive. If not, restart the agent; which will
attempt to restart supervisord via our normal startup path.
"""
if self.supervisor_psutil_process is None or self.supervisor_psutil_process.is_running():
return
LOG.warning("Supervisor failed (pid %s). Restarting agent.", self.supervisor_pid)
LOG.info("Restart agent command:%s", str(exec_args))
try:
os.execv(exec_args[0], exec_args)
except Exception, e:
LOG.exception("Failed agent re-exec: %s", e)
LOG.error("Agent exiting.")
sys.exit(-1)Then I knew the cloudera-scm-agent restarted is because the agent will check whether the process of supervisor
is running, if it's check failed, it will restart agent.
But I found that process the supervisor always running normal.So why agent check failed? The first thing is to find out how cloudera-scm-agent check.I added some code to agent.py, to see more information about supervisor when agent checked it. LIke this:
def supervisor_check(self):
"""
Check if supervisord process is still alive. If not, restart the agent; which will
attempt to restart supervisord via our normal startup path.
"""
"""Here is the added code start"""
LOG.info("Supervisor process info:%s", str(self.supervisor_psutil_process.as_dict(attrs=['pid', 'name', 'username'])))
LOG.info("Supervisor process:%s", str(self.supervisor_psutil_process))
LOG.info("Supervisor process status:%s", self.supervisor_psutil_process.status())
LOG.info("Supervisor process is_running:%s", self.supervisor_psutil_process.is_running())
LOG.info("Restart agent command:%s", str(exec_args))
"""Here is the added code stop"""
if self.supervisor_psutil_process is None or self.supervisor_psutil_process.is_running():
return
LOG.warning("Supervisor failed (pid %s). Restarting agent.", self.supervisor_pid)
LOG.info("Restart agent command:%s", str(exec_args))
try:
os.execv(exec_args[0], exec_args)
except Exception, e:
LOG.exception("Failed agent re-exec: %s", e)
LOG.error("Agent exiting.")
sys.exit(-1)The I restart the cloudera-scm-agent, and check the output of cloudera-scm-agent.log.
I found this
So check failed is because the
self.supervisor_psutil_process.is_running()
return false. The I checked the psutil source code, here it is
def _init(self, pid, _ignore_nsp=False):
if pid is None:
pid = os.getpid()
else:
if not _PY3 and not isinstance(pid, (int, long)):
raise TypeError('pid must be an integer (got %r)' % pid)
if pid < 0:
raise ValueError('pid must be a positive integer (got %s)'
% pid)
self._pid = pid
self._name = None
self._exe = None
self._create_time = None
self._gone = False
self._hash = None
# used for caching on Windows only (on POSIX ppid may change)
self._ppid = None
# platform-specific modules define an _psplatform.Process
# implementation class
self._proc = _psplatform.Process(pid)
self._last_sys_cpu_times = None
self._last_proc_cpu_times = None
# cache creation time for later use in is_running() method
try:
self.create_time()
except AccessDenied:
# we should never get here as AFAIK we're able to get
# process creation time on all platforms even as a
# limited user
pass
except NoSuchProcess:
if not _ignore_nsp:
msg = 'no process found with pid %s' % pid
raise NoSuchProcess(pid, None, msg)
else:
self._gone = True
# This pair is supposed to indentify a Process instance
# univocally over time (the PID alone is not enough as
# it might refer to a process whose PID has been reused).
# This will be used later in __eq__() and is_running().
self._ident = (self.pid, self._create_time)
def __eq__(self, other):
# Test for equality with another Process object based
# on PID and creation time.
if not isinstance(other, Process):
return NotImplemented
return self._ident == other._ident
def is_running(self):
"""Return whether this process is running.
It also checks if PID has been reused by another process in
which case return False.
"""
if self._gone:
return False
try:
# Checking if PID is alive is not enough as the PID might
# have been reused by another process: we also want to
# check process identity.
# Process identity / uniqueness over time is greanted by
# (PID + creation time) and that is verified in __eq__.
return self == Process(self.pid)
except NoSuchProcess:
self._gone = True
return FalseSo the check failed is because the process._create_time is changed,but the supervisor is running normal.
Then i guess it's the system time changed cause this, because we set the crontab mission to sync the system time every minute. We deleted the crontab mission, and the problem is fixed. And then we use the ntpd to sync
system time, the problem didn't show again.
Here is the supervisor log. I don‘t know much about the supervisor. So please help me to check it. Thanks very much.
Thank you very much for your reply.
I check all the processes. But i didn't found something wrong.
Finally, i checked the source code of the cloudera-scm-agent(/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/agent.py), and i found this
def supervisor_check(self):
"""
Check if supervisord process is still alive. If not, restart the agent; which will
attempt to restart supervisord via our normal startup path.
"""
if self.supervisor_psutil_process is None or self.supervisor_psutil_process.is_running():
return
LOG.warning("Supervisor failed (pid %s). Restarting agent.", self.supervisor_pid)
LOG.info("Restart agent command:%s", str(exec_args))
try:
os.execv(exec_args[0], exec_args)
except Exception, e:
LOG.exception("Failed agent re-exec: %s", e)
LOG.error("Agent exiting.")
sys.exit(-1)Then I knew the cloudera-scm-agent restarted is because the agent will check whether the process of supervisor
is running, if it's check failed, it will restart agent.
But I found that process the supervisor always running normal.So why agent check failed? The first thing is to find out how cloudera-scm-agent check.I added some code to agent.py, to see more information about supervisor when agent checked it. LIke this:
def supervisor_check(self):
"""
Check if supervisord process is still alive. If not, restart the agent; which will
attempt to restart supervisord via our normal startup path.
"""
"""Here is the added code start"""
LOG.info("Supervisor process info:%s", str(self.supervisor_psutil_process.as_dict(attrs=['pid', 'name', 'username'])))
LOG.info("Supervisor process:%s", str(self.supervisor_psutil_process))
LOG.info("Supervisor process status:%s", self.supervisor_psutil_process.status())
LOG.info("Supervisor process is_running:%s", self.supervisor_psutil_process.is_running())
LOG.info("Restart agent command:%s", str(exec_args))
"""Here is the added code stop"""
if self.supervisor_psutil_process is None or self.supervisor_psutil_process.is_running():
return
LOG.warning("Supervisor failed (pid %s). Restarting agent.", self.supervisor_pid)
LOG.info("Restart agent command:%s", str(exec_args))
try:
os.execv(exec_args[0], exec_args)
except Exception, e:
LOG.exception("Failed agent re-exec: %s", e)
LOG.error("Agent exiting.")
sys.exit(-1)The I restart the cloudera-scm-agent, and check the output of cloudera-scm-agent.log.
I found this
So check failed is because the
self.supervisor_psutil_process.is_running()
return false. The I checked the psutil source code, here it is
def _init(self, pid, _ignore_nsp=False):
if pid is None:
pid = os.getpid()
else:
if not _PY3 and not isinstance(pid, (int, long)):
raise TypeError('pid must be an integer (got %r)' % pid)
if pid < 0:
raise ValueError('pid must be a positive integer (got %s)'
% pid)
self._pid = pid
self._name = None
self._exe = None
self._create_time = None
self._gone = False
self._hash = None
# used for caching on Windows only (on POSIX ppid may change)
self._ppid = None
# platform-specific modules define an _psplatform.Process
# implementation class
self._proc = _psplatform.Process(pid)
self._last_sys_cpu_times = None
self._last_proc_cpu_times = None
# cache creation time for later use in is_running() method
try:
self.create_time()
except AccessDenied:
# we should never get here as AFAIK we're able to get
# process creation time on all platforms even as a
# limited user
pass
except NoSuchProcess:
if not _ignore_nsp:
msg = 'no process found with pid %s' % pid
raise NoSuchProcess(pid, None, msg)
else:
self._gone = True
# This pair is supposed to indentify a Process instance
# univocally over time (the PID alone is not enough as
# it might refer to a process whose PID has been reused).
# This will be used later in __eq__() and is_running().
self._ident = (self.pid, self._create_time)
def __eq__(self, other):
# Test for equality with another Process object based
# on PID and creation time.
if not isinstance(other, Process):
return NotImplemented
return self._ident == other._ident
def is_running(self):
"""Return whether this process is running.
It also checks if PID has been reused by another process in
which case return False.
"""
if self._gone:
return False
try:
# Checking if PID is alive is not enough as the PID might
# have been reused by another process: we also want to
# check process identity.
# Process identity / uniqueness over time is greanted by
# (PID + creation time) and that is verified in __eq__.
return self == Process(self.pid)
except NoSuchProcess:
self._gone = True
return FalseSo the check failed is because the process._create_time is changed,but the supervisor is running normal.
Then i guess it's the system time changed cause this, because we set the crontab mission to sync the system time every minute. We deleted the crontab mission, and the problem is fixed. And then we use the ntpd to sync
system time, the problem didn't show again.
