question Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability in Support Questions

Cloudera agent hearbeat failed to haproxy load balancer in high availability

Olivia-donut — Wed, 26 Nov 2025 11:04:13 GMT

Hello,

To enable Cloudera Server High Availability, I configured a HaProxy.

- Port 7183 is used for CM UI

- Port 7182 is used for cloudera agents

While CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent WARNING Port is being used by CM Agent: 7182

[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent ERROR Heartbeating to my_load_balancer:7182 failed.

Traceback (most recent call last):

File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat

response = self.requestor.request('heartbeat', heartbeat_data)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request

return self.issue_request(call_request, message_name, request_datum)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 251, in issue_request

call_response = self.transceiver.transceive(call_request)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 487, in transceive

self.write_framed_message(request)

File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 506, in write_framed_message

self.conn.request(req_method, self.req_resource, req_body, req_headers)

File "/usr/lib64/python3.11/http/client.py", line 1303, in request

self._send_request(method, url, body, headers, encode_chunked)

File "/usr/lib64/python3.11/http/client.py", line 1349, in _send_request

self.endheaders(body, encode_chunked=encode_chunked)

File "/usr/lib64/python3.11/http/client.py", line 1298, in endheaders

self._send_output(message_body, encode_chunked=encode_chunked)

File "/usr/lib64/python3.11/http/client.py", line 1097, in _send_output

self.send(chunk)

File "/usr/lib64/python3.11/http/client.py", line 1019, in send

self.sock.sendall(data)

ConnectionResetError: [Errno 104] Connection reset by peer

#######################################################

This is my Haproxy Configuration:

frontend cm-ui

bind *:7183 ssl crt /haproxy-all.key

default_backend cmservers

frontend cm-agents

bind *:7182 ssl crt /haproxy-all.key

default_backend cmagents

backend cmservers

# Main

# -----

description "BACKEND-CMSERVERS"

mode http

option forwardfor

balance roundrobin

server cm-1:7183 check ssl verify none crt /haproxy-all.key

server cm-2:7183 check backup ssl verify none crt /haproxy-all.key

backend cmagents

# Main

# -----

description "BACKEND CM-AGENTS"

mode http

option forwardfor

balance roundrobin

server cm-1:7182 check ssl verify none crt /haproxy-all.key

server cm-2:7182 check backup ssl verify none crt /haproxy-all.key

##################################################

Does someone have an idea please?

Many thanks!

Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability

upadhyayk04 — Thu, 27 Nov 2025 04:50:10 GMT

Hello @Olivia-donut

Thank you for reaching out to the Cloudera community

I hope you have followed the below document

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-high-availability.html

Sample HA conifguration file:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/managing-clusters/topics/cm-ha-proxy-sample-configuration.html

Let me know if the setup is exactly like the one mentioned in the below document

Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability

Olivia-donut — Fri, 05 Dec 2025 16:16:16 GMT

Hi,

Thank you for replying!

I have TLS enabled only for Cloudera UI, not for agents. Otherwise, I have followed the documentation.

I can't find any log from the Host Monitor side. The only log about the heartbeat is the one I provided previously from /var/log/cloudera-scm-agent/

Thank you for your help!

Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability

upadhyayk04 — Mon, 15 Dec 2025 09:11:27 GMT

Hello @Olivia-donut

Thank you for your update

Actually, the same document holds true for both scenarios, that is, with TLS as well as without TLS.

Also what about the networking angle? Let's try manual telnet

# telnet <my_load_balancer> 7182

# openssl s_client -connect my_load_balancer:7182

Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability

cjervis — Mon, 29 Dec 2025 13:36:21 GMT

@Olivia-donut Were you able to resolve the issue? If so, please mark the appropriate reply as the solution to assist others in a similar situation in the future.

Re: Cloudera agent hearbeat failed to haproxy load balancer in high availability

upadhyayk04 — Mon, 05 Jan 2026 08:55:16 GMT

Hello @Olivia-donut

Could you please confirm if you are still facing the issue?

Is the heartbeat working fine?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.