Bad agent status after adding new host

[#1]

After I added a host to the cluster, the Agent Status for the new host is in bad health. I checked the Host Agent log and I see the following errors:

[07/Jan/2016 13:40:13 +0000] 407 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused
[07/Jan/2016 13:41:12 +0000] 407 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused

I couldn't find any existing solutions to this so I appreciate any help.

Thanks,

Ali 

[#2]

Hello akashmar,

- In your cluster, are you using dns server? Infact, in any case please check host entries.

- Is your host inspector result clean? For keeping this specific, it will be nice if you are share result for this specific host.

[#3 · Accepted solution]

I found two problems:

1. The first one was the IP address of the main host. It was using 127.0.0.1 which the new host couldn't reach.

2. I had to upgrade the Cloudera Management Agent on the new host after I installed Cloudera on it, which seems weird. You'd think it would install everything with the same version as the main host.

[#4]

It generally does but it very well depends on the way we do the installation. Good to hear, your issue is resolved.

[#5]

It normally does and very-well depends on way we install them. Good to hear that issue is resolve. TC.