Hi,
I am setting up a Hadoop cluster and I am having trouble with Zookeeper health tests. Our devops team has closed all ports in the machines except those used by zookeeper, namely: 2181, 3181, and 4181. That makes the Zookeeper canary test to fail. If we open all ports in the machines, the canary test passes.
I have seen two types in problems in the logs:
WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=2, connecting to myhostname/10.2.3.23:3181
java.net.ConnectException: Connection refused.
Which is odd, because port 3181 is open.
and:
INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.2.2.11:50561
INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14ba997bb9d0e82 with negotiated timeout 30000 for client /10.2.2.11:50561
INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.2.11:50561 which had sessionid 0x14ba997bb9d0e82
WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[SERVER], CATEGORY=[LOG_MESSAGE], ROLE=[zookeeper-zk-1], SEVERITY=[IMPORTANT], SERVICE=[zookeeper], HOST_IDS=[a_hostname], SERVICE_TYPE=[ZOOKEEPER], LOG_LEVEL=[WARN], HOSTS=[a_hostname], EVENTCODE=[EV_LOG_EVENT]}, content=Got zxid 0x200000001 expected 0x1, timestamp=1424479536132}.
The ports that I mark in red change all over the logs, as if Zookeeper was trying to connect to different ports each time.
I don't know which error is causing the problem, if any.
Can you help me with this issue. Or, if you know the ports that Cloudera Manager uses to perform the zookeeper canary test, do you mind sharing them?
