Cloudera agent hearbeat failed to haproxy load balancer in high availability

[#1]

Hello, 

To enable Cloudera Server High Availability, I configured a HaProxy. 

- Port 7183 is used for CM UI

- Port 7182 is used for cloudera agents

While CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        WARNING  Port is being used by CM Agent: 7182

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        ERROR    Heartbeating to my_load_balancer:7182 failed.

Traceback (most recent call last):

  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat

    response = self.requestor.request('heartbeat', heartbeat_data)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request

    return self.issue_request(call_request, message_name, request_datum)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 251, in issue_request

    call_response = self.transceiver.transceive(call_request)

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 487, in transceive

    self.write_framed_message(request)

  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 506, in write_framed_message

    self.conn.request(req_method, self.req_resource, req_body, req_headers)

  File "/usr/lib64/python3.11/http/client.py", line 1303, in request

    self._send_request(method, url, body, headers, encode_chunked)

  File "/usr/lib64/python3.11/http/client.py", line 1349, in _send_request

    self.endheaders(body, encode_chunked=encode_chunked)

  File "/usr/lib64/python3.11/http/client.py", line 1298, in endheaders

    self._send_output(message_body, encode_chunked=encode_chunked)

  File "/usr/lib64/python3.11/http/client.py", line 1097, in _send_output

    self.send(chunk)

  File "/usr/lib64/python3.11/http/client.py", line 1019, in send

    self.sock.sendall(data)

ConnectionResetError: [Errno 104] Connection reset by peer

#######################################################

This is my Haproxy Configuration:

frontend cm-ui

    bind                        *:7183 ssl crt /haproxy-all.key

    default_backend cmservers

frontend cm-agents

    bind                        *:7182 ssl crt /haproxy-all.key

    default_backend cmagents

backend cmservers

    # Main

    # -----

    description         "BACKEND-CMSERVERS"

    mode                http

    option              forwardfor

    balance             roundrobin

    server cm-1:7183 check ssl verify none crt /haproxy-all.key

    server cm-2:7183 check backup ssl verify none crt /haproxy-all.key

backend cmagents

    # Main

    # -----

    description         "BACKEND CM-AGENTS"

    mode                http

    option              forwardfor

    balance             roundrobin

    server cm-1:7182 check ssl verify none crt /haproxy-all.key

    server cm-2:7182 check backup ssl verify none crt /haproxy-all.key

##################################################

Does someone have an idea please?

Many thanks!

[#2]

Hello @Olivia-donut 

Thank you for reaching out to the Cloudera community

I hope you have followed the below document

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-high-availability...

Sample HA conifguration file: 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-ha-proxy-sample-c...

Let me know if the setup is exactly like the one mentioned in the below document

[#3]

Hi,

Thank you for replying!

I have TLS enabled only for Cloudera UI, not for agents. Otherwise, I have followed the documentation. 

I can't find any log from the Host Monitor side. The only log about the heartbeat is the one I provided previously from /var/log/cloudera-scm-agent/

Thank you for your help!

[#4]

Hello @Olivia-donut 

Thank you for your update

Actually, the same document holds true for both scenarios, that is, with TLS as well as without TLS. 

Also what about the networking angle? Let's try manual telnet 

# telnet <my_load_balancer> 7182

# openssl s_client -connect my_load_balancer:7182

[#5]

@Olivia-donut Were you able to resolve the issue? If so, please mark the appropriate reply as the solution to assist others in a similar situation in the future.