Support Questions
Find answers, ask questions, and share your expertise

adding host agent to cluster fails

Highlighted

Re: adding host agent to cluster fails

New Contributor
Hi have you check the ntp process ? Have you try to restart the agent on this host and retry the operation ? Seems that some processes is unable to read file system information, have you try to do a df -h on this host ? There are network file system mounted ? Gianluca

Re: adding host agent to cluster fails

Explorer

   Hi! Yes, ntp works properly (I know about similary problems if ntp dont working, but looks like this error not about ntp).  We tried to restart agents and cm server (include host restart) and even try to install different version (6.1.1) of CM and agents on different hosts, but no luck. And yes, we did "df -h" on the hosts: this shows filesystems used by host. There no errors or some "strange things" in output of the command. We dont use NFS or any other network mounted FS. And I tried as example change directory to nodev fs /run/cloudera-scm-agent/process and do  "ls -la" which returns ''." and ".." whitout any delays as expected. Thus, I think, this FS used by cm-agent works properly.

    At the end, if we install agent on the same host as Cloudera Manager, then it is ok, no errors about "nodev" filesystems or any other.

Highlighted

Re: adding host agent to cluster fails

New Contributor
Thank for the information , looking at the log seems you are using the loopback interface (127.0.0.1), if so please try to set a different IP address and correct the /etc/hosts files to reflect hostname with new ip. The loopback interface can't guarantee route off the traffic out of the node. Can you try ?
Highlighted

Re: adding host agent to cluster fails

New Contributor

Can you share agent log on the host ?

BR

Highlighted

Re: adding host agent to cluster fails

Super Collaborator

Is it a new server or a server that you deleted and recreated with the same name?

try to clean the cache on the cloudera manager server service nscd restart and nscd -i hosts

Are you able to ssh to the server from the cloudera manager server?

Highlighted

Re: adding host agent to cluster fails

Explorer

Please verify your custom DHCP options as well if you are using custom DNS name hostnames.

Highlighted

Re: adding host agent to cluster fails

Explorer

same thing happened to me. I had 2 data nodes and the installation worked in one of the servers and failed with the same error on the wizard and in logs as 

 

[17/Dec/2020 23:32:42 +0000] 8744 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.02 min:0.02 mean:0.02 max:0.02 LIFE_MAX:0.02
[17/Dec/2020 23:33:49 +0000] 8744 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/user/0,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/1032

 

 

I checked my /etc/hosts. by mistake it had uppercase. I made everything to lowercase, restarted the agent. Voila ! it got installed and moved to next stage