Installation failed. Failed to receive heartbeat from agent

[#1]

Hi, 

i am trying to setup 3 node cluter using cloudera manager , getting below issue while installing

we already verified all steps given below and also check older post related to the same issue but no luck.

Installation failed. Failed to receive heartbeat from agent.
Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
Ensure that ports 9000 and 9001 are not in use on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

I don't why below image is not visible other it is public also.It is for same error above.

Starting Cloudera Manager Agent...

BEGIN /sbin/service cloudera-scm-agent status 
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent 
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled) 
Active: inactive (dead) 
Docs: man:systemd-sysv-generator(8) 
END (3) 
BEGIN /sbin/service cloudera-scm-agent start 
Starting cloudera-scm-agent (via systemctl): [ OK ] 
END (0) 
agent logs: 
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent INFO SCM Agent Version: 5.11.0 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent INFO SCM Agent Version: 5.11.0 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[20/Apr/2017 16:13:27 +0000] 4664 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
END (0) 
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' 
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory 
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory 
END (0) 
end of agent logs. 
scm agent started 
Installation script completed successfully.

all done 
closing logging file descriptor 

logs of cloudera-scm-server below

2017-04-20 16:10:36,402 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from DETECT_ROOT (PT0S) to DETECT_DISTRO
2017-04-20 16:10:36,402 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from DETECT_DISTRO (PT0S) to DETECT_SCM
2017-04-20 16:10:37,403 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from DETECT_SCM (PT1.001S) to REPO_INSTALL
2017-04-20 16:10:37,403 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from REPO_INSTALL (PT0S) to REFRESH_METADATA
2017-04-20 16:11:43,426 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from REFRESH_METADATA (PT66.023S) to PACKAGE_INSTALL jdk
2017-04-20 16:11:46,409 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from REFRESH_METADATA (PT70.021S) to PACKAGE_INSTALL jdk
2017-04-20 16:11:56,393 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from REFRESH_METADATA (PT80.027S) to PACKAGE_INSTALL jdk
2017-04-20 16:12:03,433 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from PACKAGE_INSTALL jdk (PT20.007S) to PACKAGE_INSTALL oracle-j2sdk1.7
2017-04-20 16:12:04,433 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from PACKAGE_INSTALL oracle-j2sdk1.7 (PT1S) to PACKAGE_INSTALL cloudera-manager-agent
2017-04-20 16:12:13,419 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from PACKAGE_INSTALL jdk (PT27.009S) to PACKAGE_INSTALL oracle-j2sdk1.7
2017-04-20 16:12:27,403 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from PACKAGE_INSTALL jdk (PT31.010S) to PACKAGE_INSTALL oracle-j2sdk1.7
2017-04-20 16:12:53,435 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from PACKAGE_INSTALL oracle-j2sdk1.7 (PT40.017S) to PACKAGE_INSTALL cloudera-manager-agent
2017-04-20 16:13:17,422 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from PACKAGE_INSTALL oracle-j2sdk1.7 (PT50.019S) to PACKAGE_INSTALL cloudera-manager-agent
2017-04-20 16:13:26,470 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from PACKAGE_INSTALL cloudera-manager-agent (PT82.037S) to PACKAGE_INSTALL cloudera-manager-daemons
2017-04-20 16:13:27,471 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from PACKAGE_INSTALL cloudera-manager-daemons (PT1.001S) to INSTALL_JCE
2017-04-20 16:13:27,471 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from INSTALL_JCE (PT0S) to AGENT_CONFIGURE
2017-04-20 16:13:27,471 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from AGENT_CONFIGURE (PT0S) to AGENT_START
2017-04-20 16:13:27,768 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from AGENT_START (PT0.297S) to SCRIPT_SUCCESS
2017-04-20 16:13:27,768 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Transitioning from SCRIPT_SUCCESS (PT0S) to WAIT_FOR_HEARTBEAT
2017-04-20 16:14:28,136 INFO NodeConfiguratorThread-1-2:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_masternode: Setting WAIT_FOR_HEARTBEAT as failed and done state
2017-04-20 16:14:28,136 INFO NodeConfiguratorThread-1-2:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2017-04-20 16:17:17,648 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2017-04-20 16:17:17,652 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2015-04-21T10:47:17.649Z to reap.
2017-04-20 16:17:17,654 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2017-04-20 16:17:24,541 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from PACKAGE_INSTALL cloudera-manager-agent (PT247.119S) to PACKAGE_INSTALL cloudera-manager-daemons
2017-04-20 16:17:24,542 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from PACKAGE_INSTALL cloudera-manager-daemons (PT0.001S) to INSTALL_JCE
2017-04-20 16:17:24,542 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from INSTALL_JCE (PT0S) to AGENT_CONFIGURE
2017-04-20 16:17:24,542 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from AGENT_CONFIGURE (PT0S) to AGENT_START
2017-04-20 16:17:25,510 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from AGENT_START (PT0.968S) to SCRIPT_SUCCESS
2017-04-20 16:17:25,511 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Transitioning from SCRIPT_SUCCESS (PT0.001S) to WAIT_FOR_HEARTBEAT
2017-04-20 16:17:38,559 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from PACKAGE_INSTALL cloudera-manager-agent (PT285.124S) to PACKAGE_INSTALL cloudera-manager-daemons
2017-04-20 16:17:39,560 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from PACKAGE_INSTALL cloudera-manager-daemons (PT1.001S) to INSTALL_JCE
2017-04-20 16:17:39,560 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from INSTALL_JCE (PT0S) to AGENT_CONFIGURE
2017-04-20 16:17:39,560 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from AGENT_CONFIGURE (PT0S) to AGENT_START
2017-04-20 16:17:40,371 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from AGENT_START (PT0.811S) to SCRIPT_SUCCESS
2017-04-20 16:17:40,371 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Transitioning from SCRIPT_SUCCESS (PT0S) to WAIT_FOR_HEARTBEAT
2017-04-20 16:18:25,898 INFO NodeConfiguratorThread-1-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode2: Setting WAIT_FOR_HEARTBEAT as failed and done state
2017-04-20 16:18:25,898 INFO NodeConfiguratorThread-1-1:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2017-04-20 16:18:40,748 INFO NodeConfiguratorThread-1-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: sg_slavenode1: Setting WAIT_FOR_HEARTBEAT as failed and done state
2017-04-20 16:18:40,748 INFO NodeConfiguratorThread-1-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2017-04-20 16:21:07,825 INFO 915912177@scm-web-3:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from xxx.xx.xx.xx
2017-04-20 16:27:17,676 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2017-04-20 16:27:17,680 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2015-04-21T10:57:17.676Z to reap.
2017-04-20 16:27:17,682 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2017-04-20 16:27:33,334 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.

[#2]

It is missing the cloudera agent log at /var/log/cloudera-scm-agent//cloudera-scm-agent.log

Make sure it is created and writable by the user running the CM agent process. This may not fix it but will give us the log to look at.

It also gives a warning about the /run/cloudera-scm-agent being in mode 755 instead of 751. You could try correcting that manually although I don't think this is the issue.

[#3 · Accepted solution]

I faced the same issue ...and I checked all verifcation steps suggested by cloudera

in my case the issue was I put underscore (_) character in hostname inside /etc/hosts which is not allowed.

hostname can contain only alphabets ,digits and hyphen (-) character , so if you set any special character in hostname please remove it and check it again

[#4]

Thanks shalaj.You saved my life

[#5]

매우 감사합니다. (_ _)

가장 훌륭한 답변입니다.

제 목숨도 구하셨군요!

[#6]

Of course I wondered what @KamaJinny was saying so I figured others may as well. Here is the translation via google translate: