Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error sending messages to Firehose on adding a new host

Error sending messages to Firehose on adding a new host

Expert Contributor

I am trying to add a new host located in a different data-center from where my so far single node cluster is located. 


I keep getting in cloudera-scm-agent.log and so my host is never getting live. CM keeps showing bad health and that the host was not in touch with Host monitor even though it is in contact with cloudera manager.


Please help me trouble shoot this. I came across some suggestions on this forum and would like to let you know that :

1. ping with size > 1500 bytes works:


ping -c 3 -s 1800


2. I have added the FQDN with public IP  of the existing node in /etc/hosts file of this new node I have hosted with AWS 


Please suggest what could be the problem.


Note: This new host is hosted with AWS and I have allowed communication from all ports and IP so that shouldnt be a problem.

Also, the old host is not hosted with AWS and is located somewhere else. NTPD is enabled on both the hosts and heartbeat is working fine. 


Please feel free to ask questions. Help me troubleshoot this please as it is very specific to cloudera platform and is a problem since nowhere else can I get this answered


[03/Nov/2016 15:21:55 +0000] 56464 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose: mgmt-HOSTMONITOR-5ccf80948a373fcc0e29b2976ccf7c19
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.0-py2.6.egg/cmf/monitor/", line 116, in _send
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/", line 469, in __init__
  File "/usr/lib64/python2.6/", line 742, in connect
  File "/usr/lib64/python2.6/", line 567, in create_connection
    raise error, msg
timeout: timed out



Re: Error sending messages to Firehose on adding a new host


Have you got a solution for this? If so can you please share it. I am trying to spin up a cluster using cloudera director and it fails while bootstrap of cluster nodes with above error.

Don't have an account?
Coming from Hortonworks? Activate your account here