Can one Zookeeper quorum support multiple HDFS instances for automatic HA failover?

[#1]

Is it possible use a single Zookeeper quorum for managing the HA state of more than one HDFS-HA?

If so:

1. Would there be any scaling limits with this approach? A practical upper limit on number of Namenodes that can work across a single Zookeeper quorum?
2. What steps are required to configure the multiple HDFS Namenodes to use the single Zookeeper quorum?
3. What impact would this have on upgrades and maintenance?
4. Does Hortonworks support provide support for Namenodes or Zookeeper implemented in this configuration?

In general, are there reasons why this would be not recommended?

[#2 · Accepted solution]

Looking at my current HA implementation, the root zNode for HA is hadoop-ha, followed by the name of the HA instance. IE: /hadoop-ha/HOME , where HOME is to name of the HA NN instance.

So this should support your "multiple" HA systems in theory.

HA NN doesn't but much load on ZooKeeper, so you're not going to have a scaling issue in that regard.

I think this is an operational issue though. Customers are starting to see that ZooKeeper is used for more and more things on clusters and could be considered a place that holds "too many eggs".

Another point is around upgrades. While the rolling upgrade supports an easy transition for ZooKeeper, the clusters you would serve from a single ZooKeeper instance will have limitations, from an operational standpoint.

Customers with larger clusters are trending to having multiple ZooKeepers per cluster, instead of one ZooKeeper the rule them all. Especially if Storm and Kafka are involved. They apply a different type (heavier) of load then NN HA.

Note: Resource Manager HA can also put quite a load on ZooKeeper on really large clusters, as it uses ZooKeeper to maintain the state of jobs.

[#3]

I would highly recommend against re-using another ZK quorum for this purpose. The risk of the network partitioning is too high and the benefits aren't clear. As David mentions above, NN doesn't put high load on ZK for leader election.

Have each NN HA pair (cluster for that matter) talk to their own ZK quorum within the same network segment.