Support Questions

Find answers, ask questions, and share your expertise
Announcements
Now Live: Explore expert insights and technical deep dives on the new Cloudera Community BlogsRead the Announcement

Cloudera agent hearbeat failed to haproxy load balancer in high availability

avatar
Frequent Visitor

Hello, 

To enable Cloudera Server High Availability, I configured a HaProxy. 

- Port 7183 is used for CM UI

- Port 7182 is used for cloudera agents

While CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log

 

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        WARNING  Port is being used by CM Agent: 7182
[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        ERROR    Heartbeating to my_load_balancer:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat
    response = self.requestor.request('heartbeat', heartbeat_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request
    return self.issue_request(call_request, message_name, request_datum)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 251, in issue_request
    call_response = self.transceiver.transceive(call_request)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 487, in transceive
    self.write_framed_message(request)
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 506, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python3.11/http/client.py", line 1303, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1349, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1097, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.11/http/client.py", line 1019, in send
    self.sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer
 
#######################################################
This is my Haproxy Configuration:
frontend cm-ui

    bind                        *:7183 ssl crt /haproxy-all.key
    default_backend cmservers

frontend cm-agents

    bind                        *:7182 ssl crt /haproxy-all.key
    default_backend cmagents

backend cmservers

    # Main
    # -----
    description         "BACKEND-CMSERVERS"
    mode                http
    option              forwardfor
    balance             roundrobin

    server cm-1:7183 check ssl verify none crt /haproxy-all.key
    server cm-2:7183 check backup ssl verify none crt /haproxy-all.key

backend cmagents

    # Main
    # -----
    description         "BACKEND CM-AGENTS"
    mode                http
    option              forwardfor
    balance             roundrobin

    server cm-1:7182 check ssl verify none crt /haproxy-all.key
    server cm-2:7182 check backup ssl verify none crt /haproxy-all.key
##################################################
 
Does someone have an idea please?
Many thanks!
2 REPLIES 2

avatar
Master Collaborator

Hello @Olivia-donut 

Thank you for reaching out to the Cloudera community

I hope you have followed the below document

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-high-availability...

Sample HA conifguration file: 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-ha-proxy-sample-c...

Let me know if the setup is exactly like the one mentioned in the below document

avatar
Frequent Visitor

Hi,

Thank you for replying!

I have TLS enabled only for Cloudera UI, not for agents. Otherwise, I have followed the documentation. 

I can't find any log from the Host Monitor side. The only log about the heartbeat is the one I provided previously from /var/log/cloudera-scm-agent/

Thank you for your help!