Reply
Explorer
Posts: 6
Registered: ‎02-25-2019

adding host agent to cluster fails

[ Edited ]

After using Cloudera wizard to add host to cluster ... the slave agent logs show the error below

[07/Mar/2019 18:54:41 +0000] 20033 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[07/Mar/2019 18:54:43 +0000] 20033 MainThread supervisor   INFO     Trying to connect to supervisor (Attempt 1)
[07/Mar/2019 18:54:43 +0000] 20033 MainThread supervisor   INFO     Supervisor version: 3.0, pid: 18803
[07/Mar/2019 18:54:43 +0000] 20033 MainThread supervisor   INFO     Successfully connected to supervisor
[07/Mar/2019 18:54:43 +0000] 20033 MainThread agent        INFO     Supervisor version: 3.0, pid: 18803
[07/Mar/2019 18:54:43 +0000] 20033 MainThread agent        INFO     Connecting to previous supervisor: agent-18803-1551977586.
[07/Mar/2019 18:54:45 +0000] 20033 MainThread supervisor   INFO     Triggering supervisord update.
[07/Mar/2019 18:54:45 +0000] 20033 MainThread _cplogging   INFO     [07/Mar/2019:18:54:45] ENGINE Bus STARTING
[07/Mar/2019 18:54:45 +0000] 20033 MainThread _cplogging   INFO     [07/Mar/2019:18:54:45] ENGINE Started monitor thread '_TimeoutMonitor'.
[07/Mar/2019 18:54:45 +0000] 20033 MainThread _cplogging   INFO     [07/Mar/2019:18:54:45] ENGINE Serving on http://127.0.0.1:9001
[07/Mar/2019 18:54:45 +0000] 20033 MainThread _cplogging   INFO     [07/Mar/2019:18:54:45] ENGINE Bus STARTED
[07/Mar/2019 18:54:45 +0000] 20033 MainThread daemon       INFO     New monitor: (<cmf.monitor.host.HostMonitor object at 0x7f58b56cdb10>,)
[07/Mar/2019 18:54:45 +0000] 20033 MonitorDaemon-Scheduler daemon       INFO     Monitor ready to report: ('HostMonitor',)
[07/Mar/2019 18:54:45 +0000] 20033 MainThread agent        INFO     Setting default socket timeout to 45
[07/Mar/2019 18:54:45 +0000] 20033 MainThread agent        INFO     Previously active parcels: {'SPARK2': '2.3.0.cloudera4-1.cdh5.13.3.p0.611179', 'CDH': '5.14.4-1.cdh5.14.4.p0.3'}
[07/Mar/2019 18:54:45 +0000] 20033 MainThread agent        INFO     Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
[07/Mar/2019 18:54:45 +0000] 20033 Monitor-HostMonitor network_interfaces INFO     NIC iface virbr0 doesn't support ETHTOOL (95)
[07/Mar/2019 18:54:45 +0000] 20033 MainThread heartbeat_tracker INFO     HB stats (seconds): num:1 LIFE_MIN:0.02 min:0.02 mean:0.02 max:0.02 LIFE_MAX:0.02
[07/Mar/2019 18:55:52 +0000] 20033 Monitor-HostMonitor throttling_logger ERROR    Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/user/1000,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0

and CM wizard displays Error message

Failed to receive heart beat from agent

Capture.JPG 

Expert Contributor
Posts: 92
Registered: ‎01-08-2016

Re: adding host agent to cluster fails

Hello @Exor,

 

Lets step back a bit. To understand the issue better, I would request you to please help with couple of pre-lims:

  • Is agent service installed and agent log is clean?
  • Is connectivity from agent to server on configured port working?

Hope this gives more clarity on next steps.

Explorer
Posts: 6
Registered: ‎02-25-2019

Re: adding host agent to cluster fails

[ Edited ]