Support Questions

Find answers, ask questions, and share your expertise

Cloudera agent hearbeat failed to haproxy load balancer in high availability

avatar

Hello, 

To enable Cloudera Server High Availability, I configured a HaProxy. 

- Port 7183 is used for CM UI

- Port 7182 is used for cloudera agents

While CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log

 

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        WARNING  Port is being used by CM Agent: 7182
[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        ERROR    Heartbeating to my_load_balancer:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat
    response = self.requestor.request('heartbeat', heartbeat_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request
    return self.issue_request(call_request, message_name, request_datum)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 251, in issue_request
    call_response = self.transceiver.transceive(call_request)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 487, in transceive
    self.write_framed_message(request)
  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 506, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python3.11/http/client.py", line 1303, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1349, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.11/http/client.py", line 1097, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.11/http/client.py", line 1019, in send
    self.sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer
 
#######################################################
This is my Haproxy Configuration:
frontend cm-ui

    bind                        *:7183 ssl crt /haproxy-all.key
    default_backend cmservers

frontend cm-agents

    bind                        *:7182 ssl crt /haproxy-all.key
    default_backend cmagents

backend cmservers

    # Main
    # -----
    description         "BACKEND-CMSERVERS"
    mode                http
    option              forwardfor
    balance             roundrobin

    server cm-1:7183 check ssl verify none crt /haproxy-all.key
    server cm-2:7183 check backup ssl verify none crt /haproxy-all.key

backend cmagents

    # Main
    # -----
    description         "BACKEND CM-AGENTS"
    mode                http
    option              forwardfor
    balance             roundrobin

    server cm-1:7182 check ssl verify none crt /haproxy-all.key
    server cm-2:7182 check backup ssl verify none crt /haproxy-all.key
##################################################
 
Does someone have an idea please?
Many thanks!
1 REPLY 1

avatar
Master Collaborator

Hello @Olivia-donut 

Thank you for reaching out to the Cloudera community

I hope you have followed the below document

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-high-availability...

Sample HA conifguration file: 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-ha-proxy-sample-c...

Let me know if the setup is exactly like the one mentioned in the below document