question Re: Zookeeper Service showing "bad health" after deployment in Support Questions

Zookeeper Service showing "bad health" after deployment

Seaport — Mon, 01 Jun 2026 17:15:18 GMT

I just finished installing a new cluster but failed to start the zookeeper service. Each instance started but had the "bad health" flag. The zookeeper log on node 3 showed this error repeatedly.

++++
Cannot open channel to 1 at election address node1/61.62.63.1:4181
java.net.ConnectException: Connection refused
++++

On node 3, the connection to node 1 zookeeper port showed no issue.

$ nc -zv node1 4181
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 61.62.63.1:4181.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

On node1, the zookeeper port is open too
$ ss -tulnp | grep 4181
tcp LISTEN 0 50 61.62.63.1:4181 0.0.0.0:*

The private cloud base cluster is 7.1.9 sp1.

Thank you.

Regards,

Re: Zookeeper Service showing "bad health" after deployment

DianaTorres — Mon, 01 Jun 2026 21:54:23 GMT

@Seaport Hello team @pajoshi @shubham_sharma
Do you have any insights here? Thanks!

Re: Zookeeper Service showing "bad health" after deployment

rki_ — Tue, 02 Jun 2026 07:38:39 GMT

Hello @Seaport ,

Can you check if the reverse DNS lookup is working fine from each zookeeper host. Also worth checking the zoo.cfg file under the latest /var/run/cloudera-scm-agent/process/ xxxx-zookeeper-server/ directory to check how the servers and ports are configured.

Re: Zookeeper Service showing "bad health" after deployment

Seaport — Tue, 02 Jun 2026 17:40:01 GMT

Cloudera Support helped me resolved this issue.

* The "Bad Health" status displayed in Cloudera Manager was a false-positive monitoring alert.

* The Cloudera Manager Service Monitor (SMON) was failing its secure TLS connection handshakes to ZooKeeper due to strict endpoint identification checks introduced in modern Java runtimes (Java 17). Because SMON couldn't pull health metrics, it flagged ZooKeeper as down.

* The solution is to configure the JVM argument inside the SMON configuration "Java Configuration Options for Service Monitor (firehose_java_opts)" to bypass the strict certificate hostname checks: `-Djdk.rmi.ssl.client.enableEndpointIdentification=false`.

This cluster is 7.1.9sp1 with 7.13.1 CM. Strangely, another cluster, which has the same cluster version, CM version, and Java version, had no such issue. It was set up six months ago.

Thanks for all the responses.

Best regards,