Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager agent got restarted automatically

Cloudera Manager agent got restarted automatically

Hi All,

 

Is anyone aware if cloudera manager agent can get restarted automatically?

Is there any process on Linux that can trigger this restart?

3 REPLIES 3
Highlighted

Re: Cloudera Manager agent got restarted automatically

Expert Contributor

Hello - welcome to the Community!

 

To give you the best answer we'd need to know what kind of symptoms you observed, or generally what is leading you to this line of investigation.

 

Can you share a tiny portion of /var/log/cloudera-scm-agent/cloudera-scm-agent.log from just before and just after the agent was restarted? How many hosts had the agent restart - was it just one, or several/all? Did the agent itself restart or did you experience any other interruption in any other service at the same time?

 

Thanks,

Highlighted

Re: Cloudera Manager agent got restarted automatically

New Contributor

Hi Team,

 

We faced a similar issue, restart of cloudera agent and consequently datanode and Node manager. Please find the logs for reference.the restart happened @ 24-FEB 1:48 AM PST

 

[23/Feb/2016 22:26:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants system CPU lower than expected for process 119327: 34149.12, 30874.67
[23/Feb/2016 23:26:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants user CPU lower than expected for process 119327: 415380.88, 162453.03
[23/Feb/2016 23:26:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants system CPU lower than expected for process 119327: 34196.82, 30923.4
[24/Feb/2016 00:26:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants user CPU lower than expected for process 119327: 416520.88, 162969.17
[24/Feb/2016 00:26:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants system CPU lower than expected for process 119327: 34354.19, 31034.41
[24/Feb/2016 01:27:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (60 skipped) Descendants user CPU lower than expected for process 119327: 416998.23, 163139.15
[24/Feb/2016 01:27:46 +0000] 2997 MonitorDaemon-Reporter throttling_logger INFO (60 skipped) Descendants system CPU lower than expected for process 119327: 34390.86, 31073.72
[24/Feb/2016 01:48:44 +0000] 3134 MainThread agent INFO SCM Agent Version: 5.3.3
[24/Feb/2016 01:48:44 +0000] 3134 MainThread agent INFO Agent Protocol Version: 4
[24/Feb/2016 01:48:44 +0000] 3134 MainThread agent INFO Using Host ID: 29fa5b47-6526-4794-bb4e-45754d8d39e4
[24/Feb/2016 01:48:44 +0000] 3134 MainThread agent INFO Using directory: /var/run/cloudera-scm-agent
[24/Feb/2016 01:48:44 +0000] 3134 MainThread agent INFO Using supervisor binary path: /usr/lib64/cmf/agent/src/cmf/../../build/env/bin/supervisord

Highlighted

Re: Cloudera Manager agent got restarted automatically

Super Guru

I would check your /var/log/messages and/or your last log to check if the host rebooted.

Even if the the agent AND the supervisord crashed, they would not restart.  They would only start themselves if the init.d was run.

Most likely you had the host reboot which took everything down hard.

 

Don't have an account?
Coming from Hortonworks? Activate your account here