Hi guys

This is not an installation issue for me. This cluster I setup here is running 24X7 for 2 years ! 

In my CM managed CDH 5.8.0 hadoop cluster I am getting this error on one datanode and this error I used to get in 5.6.0 and now in 5.8.0 ( I thought this error may go away after I move to 5.8.0)

1. From this datanode if I telnet, its actually successful

    telnet xx.xxx.xx.xxx 7182

    Tryingxx.xxx.xx.xxx...

    Connected toxx.xxx.xx.xxx.

     Escape character is '^]'.

2. Also if I go to http://namenode_ip:50070 then I can see this datanode up and ready. None of the services on this node are really disrupted. But somehow cm-agent is not able to talk to cm-server ? 

[28/Jul/2016 00:38:41 +0000] 1200 MainThread agent        ERROR    Heartbeating to xx.xxx.xx.xxx:7182 failed.

Traceback (most recent call last):

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 1211, in _send_heartbeat

    response = self.requestor.request('heartbeat', dict(request=heartbeat))

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 136, in request

    self.write_call_request(message_name, request_datum, buffer_encoder)

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 178, in write_call_request

    self.write_request(message.request, request_datum, encoder)

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 182, in write_request

    datum_writer.write(request_datum, encoder)

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/io.py", line 768, in write