Heartbeating to namenode failed from datanode

[#1]

[18/Apr/2018 07:05:11 +0000] 1994 MainThread agent WARNING Long HB processing time: 45.0500581264[18/Apr/2018 07:05:11 +0000] 1994 MainThread agent WARNING Delayed HB: 40s since last[18/Apr/2018 07:05:56 +0000] 1994 MainThread agent ERROR Heartbeating to instance-1.c.phrasal-academy-200708.internal:7182 failed.Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/agent.py", line 1412, in _send_heartbeat self.master_port) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__ self.conn.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise errtimeout: timed out

Please suggest where exactly the problem is?

I have checked 

/etc/hosts and /etc/cloudera-scm-agent/config.ini multiple times

[#2]

There are various possible issues for that.

Have you checked connectivity from agent node to cloudera manager? The agent connects to CM on port 7182.

Have you tested that you get a reply and there is no network or firewall issue?

[#3]

@Apoorva06

Have you disabled SELinux? if not, pls do it, it may help you

[#4]

Thanks guys for your response....

I tried re-installing the setup but this time I notice something new in scm-server logs.

2018-04-20 07:35:52,040 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands2018-04-20 07:35:52,056 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2016-04-20T07:35:52.041Z to reap.2018-04-20 07:35:52,057 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators

I am not able to open web UI of cloudera manager,it says connection time out 

Response to curl to scm-server IP

curl: (7) Failed connect to 35.196.xx.xxx:7180; Connection timed out

netstat -anp | grep 7180

tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 17743/java

Apart from that there is no ERROR in log file,did a grep as well

Last few lines of Log:-

2018-04-20 07:26:03,921 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2018-04-20T07:26:03.921Z
2018-04-20 07:26:05,930 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: AgentAvroServlet: heartbeat processing stats: average=6ms, min=6ms, max=6ms.
2018-04-20 07:26:05,938 INFO 805264547@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=26ms, min=26ms, max=26ms.
2018-04-20 07:26:05,938 INFO 805264547@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=0ms.
2018-04-20 07:26:05,982 INFO avro-servlet-hb-processor-1:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=2, hostId=64445efb-2a18-4ffb-b6cd-0095745a5d34, hostName=namenode.c.hopeful-nova-200813.internal}] Added to rack group: /default
2018-04-20 07:26:06,499 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.
2018-04-20 07:26:21,655 INFO avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=3, hostId=3be8bdb3-1165-4b95-b4b4-c1f2988a999e, hostName=datanode2.c.hopeful-nova-200813.internal}] Added to rack group: /default
2018-04-20 07:26:42,600 INFO avro-servlet-hb-processor-3:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=1, hostId=dffd2a15-bc4e-4406-bbea-4c44d5c1fad0, hostName=datanode1.c.hopeful-nova-200813.internal}] Added to rack group: /default
2018-04-20 07:27:05,939 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=10ms, min=6ms, max=40ms.
2018-04-20 07:28:05,988 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=8ms, min=5ms, max=40ms.
2018-04-20 07:29:06,009 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=5ms, max=40ms.
2018-04-20 07:30:06,066 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=5ms, max=40ms.
2018-04-20 07:31:06,110 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=5ms, max=40ms.
2018-04-20 07:32:06,159 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:33:06,196 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:34:06,239 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:35:06,292 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:35:52,040 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2018-04-20 07:35:52,056 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2016-04-20T07:35:52.041Z to reap.
2018-04-20 07:35:52,057 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2018-04-20 07:36:06,345 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:36:06,346 INFO 805264547@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=7ms, min=1ms, max=43ms.
2018-04-20 07:36:06,346 INFO 805264547@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=1ms.
2018-04-20 07:37:06,394 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:38:06,431 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.
2018-04-20 07:39:06,484 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=5ms, max=40ms.