Skip to main content
javang
Gold ⭐️
Feb 23, 2015 7:07 PM UTC
Solved

Zookeeper Canary Test Failures: Required Ports for Cloudera Manager

Hi,

I am setting up a Hadoop cluster and I am having trouble with Zookeeper health tests. Our devops team has closed all ports in the machines except those used by zookeeper, namely: 2181, 3181, and 4181. That makes the Zookeeper canary test to fail. If we open all ports in the machines, the canary test passes.

I have seen two types in problems in the logs:

WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=2, connecting to myhostname/10.2.3.23:3181
java.net.ConnectException: Connection refused.

Which is odd, because port 3181 is open.

and:

INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.2.2.11:50561

INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14ba997bb9d0e82 with negotiated timeout 30000 for client /10.2.2.11:50561
INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.2.11:50561 which had sessionid 0x14ba997bb9d0e82
WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[SERVER], CATEGORY=[LOG_MESSAGE], ROLE=[zookeeper-zk-1], SEVERITY=[IMPORTANT], SERVICE=[zookeeper], HOST_IDS=[a_hostname], SERVICE_TYPE=[ZOOKEEPER], LOG_LEVEL=[WARN], HOSTS=[a_hostname], EVENTCODE=[EV_LOG_EVENT]}, content=Got zxid 0x200000001 expected 0x1, timestamp=1424479536132}.

The port (50561) changes all over the logs, as if Zookeeper was trying to connect to different ports each time.

I don't know which error is causing the problem, if any.

Can you help me with this issue. Or, if you know the ports that Cloudera Manager uses to perform the zookeeper canary test, do you mind sharing them?

javang
javangAnswer
Gold ⭐️
Feb 23, 2015 9:18 PM UTC

 

After some research I found the solution:

 

http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_ig_ports_cdh5.html

 

The Cloudera Manager machine must be able to connect to all the ports of a zookeeper server

 

 

2 replies

javang
javangAuthorAnswer
Gold ⭐️
Feb 23, 2015 9:18 PM UTC

 

After some research I found the solution:

 

http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2-x/topics/cdh_ig_ports_cdh5.html

 

The Cloudera Manager machine must be able to connect to all the ports of a zookeeper server

 

 

Translate
Clint
Platinum ⭐️
Platinum ⭐️
Feb 26, 2015 9:14 PM UTC

@javang thank you for closing the loop with us and posting your findings!

Translate