question Re: Installation failed. Failed to receive heartbeat from agent. in Support Questions

Installation failed. Failed to receive heartbeat from agent.

enelso — Mon, 18 Nov 2019 14:09:18 GMT

I am trying to install Cloudrea Manager Standard edition and CHD4 with parcels. This is being installed on three Dell machines running Ubuntu 12.04.2 LTS 64 bit. I am receiving an error on all three machines:

Ensure that the host's hostname is configured properly.

Ensure that port 7182 is accessible on the Cloudera Manager server (check firewall rules).

Ensure that ports 9000 and 9001 are free on the host being added.

Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

I checked that the hostname is configured.

I checked that Ports 7182, 9000 and 9001 are free (I am guessing that Cloudera is using 9000 & 9001 for python because these ports are in use after the install fails but not before the install).

sudo netstat -tulpn

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 1104/X

tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 1123/dnsmasq

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 655/sshd

tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 792/cupsd

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 29403/0

tcp6 0 0 :::6000 :::* LISTEN 1104/X

tcp6 0 0 :::22 :::* LISTEN 655/sshd

tcp6 0 0 ::1:631 :::* LISTEN 792/cupsd

tcp6 0 0 ::1:6010 :::* LISTEN 29403/0

udp 0 0 127.0.0.1:53 0.0.0.0:* 1123/dnsmasq

udp 0 0 0.0.0.0:68 0.0.0.0:* 1073/dhclient

udp 0 0 0.0.0.0:36051 0.0.0.0:* 790/avahi-daemon: r

udp 0 0 0.0.0.0:5353 0.0.0.0:* 790/avahi-daemon: r

udp6 0 0 :::50807 :::* 790/avahi-daemon: r

udp6 0 0 :::5353 :::* 790/avahi-daemon: r

And I checked the firewalls and they are open

sudo iptables -L

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain FORWARD (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

Lastly I checked the log files in /var/log/cloudera-scm-agent/ and they do not show any errors and there was only one warning, that the default socket timeout was set to 30.

Can anyone point me to what the possible problem is? We are looking at using Hadoop in one of our solutions and are trying to evaluate it before purchasing the Enterprise version. I cannot use a cloud version because of data restrictions put on my by the data vendor and client so I need to have an internal sandbox to get an idea of what we need to develop and what we will need to support. Thanks!

Re: Installation failed. Failed to receive heartbeat from agent.

dvohra — Wed, 21 Aug 2013 16:30:37 GMT

What is the error message?

Re: Installation failed. Failed to receive heartbeat from agent.

dvohra — Wed, 21 Aug 2013 16:38:45 GMT

Update /etc/hosts of all the nodes in the cluster according to the following format for each entry.

<IP address> <FQDN> <shortname>

Re: Installation failed. Failed to receive heartbeat from agent.

dvohra — Wed, 21 Aug 2013 16:40:55 GMT

After /etc/hosts files have been updated, restart the cloudera-scm-agent on each node to apply the changes of the /etc/hosts file.

Re: Installation failed. Failed to receive heartbeat from agent.

enelso — Wed, 21 Aug 2013 18:51:23 GMT

There error was Installation failed. Failed to receive heartbeat from agent.

The service was already stopped. I restarted it and still recieved the Failed to recieve hearbeat error.

sudo service cloudera-scm-agent restart
cloudera-scm-agent is already stopped
Starting cloudera-scm-agent: * cloudera-scm-agent started

Re: Installation failed. Failed to receive heartbeat from agent.

enelso — Wed, 21 Aug 2013 18:54:21 GMT

In the /etc/hostsfile I have it configured as:

127.0.0.1 localhost enelso-cluster-001

And the /etc/hostname file is

enelso-cluster-001

Re: Installation failed. Failed to receive heartbeat from agent.

dvohra — Wed, 21 Aug 2013 19:58:23 GMT

Not the local host entry.

Find the IP address, FQDN and shortname for each of nodes and add them to the /etc/hosts on all nodes in the format:

To get shortname and FQDN:

>hostname && hostname -f

Re: Installation failed. Failed to receive heartbeat from agent.

Clint — Wed, 21 Aug 2013 20:46:25 GMT

@enelso: Your hostname needs to be tied to an actual IP address on your local network which can send/receive traffic between all the hosts. The address you have associated your hostname with is the loopback address, which cannot route actual network traffic off the host.

Use "ifconfig -a" to see a listing of your network interfaces and choose one that has an actual IP address.

Re: Installation failed. Failed to receive heartbeat from agent.

enelso — Thu, 22 Aug 2013 15:18:32 GMT

@clint, DUH!!!! I can't believe I did that. I was seeing the entire IP in my head everytime I looked at that! I already knew the IP's because I had to enter them in an earlier step in the wizard configuration. Thanks for seeing my error!

Re: Installation failed. Failed to receive heartbeat from agent.

Clint — Thu, 22 Aug 2013 17:19:33 GMT

We all fall victim to that, glad to be of assistance!

Re: Installation failed. Failed to receive heartbeat from agent.

awenclaw — Mon, 16 Dec 2013 23:40:31 GMT

Hello!

I bealive I have similar (or identical) problem but I don't fully understand explanation. I'm installing cluster on two virtual machines with Ubuntu 12.04 on them. Both have two network interfaces eth0 and eth1. eth0 is connection with internet through NAT and eth1 is virtual network between them.

Host 1 name is Hadoop1.local and its IP (eth1) is: 192.168.56.101.

Host 2 name is Hadoop2.local and its IP (eth1) is: 192.168.56.102

I start Cloudera Manager instalator on Hadoop1. In step "" I type both IP addresses (.101 & .102) (Is it OK or should I only start installation on second host- Hadoop2). During "Cluster Installation" step on Hadoop1 I get error:

"Installation failed. Failed to receive heart beat from agent".

I check ports 7182, 9000, 9001 and those are OK (before I start installation those are empty, after installation I have python on them).

In /var/log/cloudera-scm-agent I see no errors.

So I assume there is something wrong with my hosts names.

My /etc/host looks like this (Hadoop1):

127.0.0.1 localhost loopback

192.168.56.102 Hadoop2.local

And second /etc/host like this (Hadoop2):

127.0.0.1 localhost loopback

192.168.56.101 Hadoop1.local

Any help would be appreciate!

Regards

Andrzej

Re: Installation failed. Failed to receive heartbeat from agent.

awenclaw — Thu, 19 Dec 2013 12:56:53 GMT

Hello!

Any idea how to solve above problem? Any help would be appreciate!

Regards

Andrzej

Re: Installation failed. Failed to receive heartbeat from agent.

Clint — Thu, 19 Dec 2013 16:52:48 GMT

Hello,

I apologize for the delay. I think there may be a couple of things going on. For starters, you should add your own hostname and IP address to the /etc/hosts file on each machine. In other words, both Host1 and Host2 entries should be in the /etc/hosts file on both machines. Also, have you checked to see if iptables is running? That is a firewall app that can stop traffic between nodes. To identify if iptables is running and disable it, do this (as root):

$ sudo chkconfig iptables --list
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

If you see iptables as "on" for any of those values (especially #5), than it's probably getting in your way, you should disable it unless you've got a company policy requiring it be enabled. To disable it, run this command:

sudo chkconfig iptables off

Re: Installation failed. Failed to receive heartbeat from agent.

awenclaw — Sun, 22 Dec 2013 22:13:25 GMT

Hi,

Thank You for Your response. I think I had something wrong in my /etc/hosts. Also on one node I had ufw enabled. Following command do the work:

sudo ufw disable

Thank You for Your support!

Regards

Andrzej

Re: Installation failed. Failed to receive heartbeat from agent.

mukul — Wed, 25 Jun 2014 23:47:37 GMT

I am facing the same exact problem not sure if it is related or not but seems very very close.

I am trying to install CDH5 on a single node cluster. .. so everything is supposed to be running on the same host ( cloudera manager, and all other hadoop and spark services), more of a sandbox than anything else,

I am running ubuntu 64 bit precise OS.

I am using cloudera manager admin to install and and cluster installation step it fails:

Installation failed. Failed to receive heartbeat from agent.

Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
Ensure that ports 9000 and 9001 are free on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

Thanks,

Mukul

Re: Installation failed. Failed to receive heartbeat from agent.

rajgoaj — Sat, 25 Oct 2014 21:28:54 GMT

@clint, I tried picking the IP from ifconfig -a command and added that to hosts file as below

I used hostname and hostname -f to find the shortname and FQDN. I still get the following error during installation

Installation failed. Failed to receive heartbeat from agent.

Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager server (check firewall rules).
Ensure that ports 9000 and 9001 are free on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

I have enabled all TCP ports and ICMP ports. I am not sure what am I missing. Could you please help should I be checking something else?

Re: Installation failed. Failed to receive heartbeat from agent.

anils369 — Mon, 17 Aug 2015 06:19:49 GMT

Hi,

I'm exactl facing the same issue:

can some one please help me to resolve this issue, I have following information in my /etc/hosts file

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.42.101 n1.example.com n1

192.168.42.102 n2.example.com n2

192.168.42.103 n3.example.com n3

I can see successful installation message in n1 and above failure message in rest of two nodes.

Please help me

Re: Installation failed. Failed to receive heartbeat from agent.

dongchen — Tue, 17 May 2016 23:44:49 GMT

I also have this issue - 'Failed to receive heartbeat from agent.' I use Cloudera Manager Portal to add one more node to the cluster and use package installation and assign roles on this node. This issue happened in the phrase when the Cloudera Manager Agent has started.

What happend is :

1. the cloudera manager agent has started, the cloudera-manager-server can detect the new host

2. the heartbeat can't be reached by cloudera-scm-agent

Can you explain a little bit details about the how the heartbeat happen betwwen Cloudera Manager Server and Cloudera Manager Agent?

How can the cloudera-scm-manager send the command like 'yum install' 'yum list' on the new node? And what port does cloudear-scm-server receive heartbeat from cloudera-scm-agent? What does cloudera-scm-agent do on the port 9000? Who will start supervisord and how does it start it?

Re: Installation failed. Failed to receive heartbeat from agent.

sim6 — Sat, 05 Nov 2016 18:04:29 GMT

Here's a step by step guide to troubleshoot this error:

http://www.yourtechchick.com/hadoop/failed-receive-heartbeat-agent-cloudera-hadoop/

Re: Installation failed. Failed to receive heartbeat from agent.

robi — Mon, 12 Jun 2017 00:45:57 GMT

Clint, my hosts file look like this, and seemed it is not working.
52.60.239.41 ip-172-31-12-19.ca-central-1.compute.internal
52.60.157.176 ip-172-31-3-218.ca-central-1.compute.internal
35.182.18.189 ip-172-31-3-179.ca-central-1.compute.internal
35.182.25.52 ip-172-31-2-137.ca-central-1.compute.internal
do I have to give a short name at end, where this shortname come from my EC2 instance name, right?