Ports used when doing zookeeper canary test

javang — Fri, 16 Sep 2022 09:22:26 GMT

Hi,

I am setting up a Hadoop cluster and I am having trouble with Zookeeper health tests. Our devops team has closed all ports in the machines except those used by zookeeper, namely: 2181, 3181, and 4181. That makes the Zookeeper canary test to fail. If we open all ports in the machines, the canary test passes.

I have seen two types in problems in the logs:

WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=2, connecting to myhostname/10.2.3.23:3181
java.net.ConnectException: Connection refused.

Which is odd, because port 3181 is open.

and:

INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.2.2.11:50561

INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14ba997bb9d0e82 with negotiated timeout 30000 for client /10.2.2.11:50561
INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.2.11:50561 which had sessionid 0x14ba997bb9d0e82
WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[SERVER], CATEGORY=[LOG_MESSAGE], ROLE=[zookeeper-zk-1], SEVERITY=[IMPORTANT], SERVICE=[zookeeper], HOST_IDS=[a_hostname], SERVICE_TYPE=[ZOOKEEPER], LOG_LEVEL=[WARN], HOSTS=[a_hostname], EVENTCODE=[EV_LOG_EVENT]}, content=Got zxid 0x200000001 expected 0x1, timestamp=1424479536132}.

The ports that I mark in red change all over the logs, as if Zookeeper was trying to connect to different ports each time.

I don't know which error is causing the problem, if any.

Can you help me with this issue. Or, if you know the ports that Cloudera Manager uses to perform the zookeeper canary test, do you mind sharing them?

Re: Ports used when doing zookeeper canary test

javang — Mon, 23 Feb 2015 21:18:34 GMT

After some research I found the solution:

http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_ig_ports_cdh5.html

The Cloudera Manager machine must be able to connect to all the ports of a zookeeper server

Re: Ports used when doing zookeeper canary test

Clint — Thu, 26 Feb 2015 21:14:56 GMT

@javang thank you for closing the loop with us and posting your findings!

question Re: Ports used when doing zookeeper canary test in Archives of Support Questions (Read Only)

Ports used when doing zookeeper canary test

Re: Ports used when doing zookeeper canary test

Re: Ports used when doing zookeeper canary test