@rseidler,
Since the basics are covered, I'll say that the stack trace you provided looks pretty odd and indicates that the agent was reading a reply from Cloudera Manager but before it could complete, the connection went away...
The Error-Log of the agent is:
>>[20/Sep/2018 10:25:06 +0000] 9377 MainThread agent ERROR Heartbeating to node001:7182 failed.
>>Traceback (most recent call last):
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1371, in _send_heartbeat
>> response = self.requestor.request('heartbeat', heartbeat_data)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
>> return self.issue_request(call_request, message_name, request_datum)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
>> call_response = self.transceiver.transceive(call_request)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
>> result = self.read_framed_message()
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 491, in read_framed_message
>> framed_message = response_reader.read_framed_message()
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 411, in read_framed_message
>> buffer_length = self._read_buffer_length()
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 424, in _read_buffer_length
>> raise ConnectionClosedException("Reader read 0 bytes.")
>>ConnectionClosedException: Reader read 0 bytes.
We see in the code that this means that no bytes were received:
421 def _read_buffer_length(self):
422 read = self.reader.read(BUFFER_HEADER_LENGTH)
423 if read == '':
424 raise ConnectionClosedException("Reader read 0 bytes.")
This does not appear to be a TCP problem, so I would assert that we will likely find some more information on the Cloudera Manager side.
Please check:
/var/log/cloudera-scm-server/cloudera-scm-server.log
See if you find messages regarding that host or regarding a problem processing heartbeats.
Since this is a "clean" problem, I suspect that Cloudera Manager may not be accepting the heartbeat and should hopefully tell you why.
*** NOTE: If Cloudera doesn't show any information, check to see what server is listening on port 7182 just to make sure it is really CM:
# netstat -nap |grep 7182 |grep LISTEN
(note the pid)
# ps aux |grep <pid>
For example:
# netstat -nap |grep 7182|grep LISTEN
tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 28669/java
# ps aux |grep 28669 |grep "cmf.Main"
This should return one result which is the CM process.
