Created 09-25-2015 09:19 PM
Is it possible use a single Zookeeper quorum for managing the HA state of more than one HDFS-HA?
If so:
In general, are there reasons why this would be not recommended?
Created 09-26-2015 02:25 AM
Looking at my current HA implementation, the root zNode for HA is hadoop-ha, followed by the name of the HA instance. IE: /hadoop-ha/HOME , where HOME is to name of the HA NN instance.
So this should support your "multiple" HA systems in theory.
HA NN doesn't but much load on ZooKeeper, so you're not going to have a scaling issue in that regard.
I think this is an operational issue though. Customers are starting to see that ZooKeeper is used for more and more things on clusters and could be considered a place that holds "too many eggs".
Another point is around upgrades. While the rolling upgrade supports an easy transition for ZooKeeper, the clusters you would serve from a single ZooKeeper instance will have limitations, from an operational standpoint.
Customers with larger clusters are trending to having multiple ZooKeepers per cluster, instead of one ZooKeeper the rule them all. Especially if Storm and Kafka are involved. They apply a different type (heavier) of load then NN HA.
Note: Resource Manager HA can also put quite a load on ZooKeeper on really large clusters, as it uses ZooKeeper to maintain the state of jobs.
Created 09-26-2015 02:25 AM
Looking at my current HA implementation, the root zNode for HA is hadoop-ha, followed by the name of the HA instance. IE: /hadoop-ha/HOME , where HOME is to name of the HA NN instance.
So this should support your "multiple" HA systems in theory.
HA NN doesn't but much load on ZooKeeper, so you're not going to have a scaling issue in that regard.
I think this is an operational issue though. Customers are starting to see that ZooKeeper is used for more and more things on clusters and could be considered a place that holds "too many eggs".
Another point is around upgrades. While the rolling upgrade supports an easy transition for ZooKeeper, the clusters you would serve from a single ZooKeeper instance will have limitations, from an operational standpoint.
Customers with larger clusters are trending to having multiple ZooKeepers per cluster, instead of one ZooKeeper the rule them all. Especially if Storm and Kafka are involved. They apply a different type (heavier) of load then NN HA.
Note: Resource Manager HA can also put quite a load on ZooKeeper on really large clusters, as it uses ZooKeeper to maintain the state of jobs.
Created 09-28-2015 09:45 PM
I would highly recommend against re-using another ZK quorum for this purpose. The risk of the network partitioning is too high and the benefits aren't clear. As David mentions above, NN doesn't put high load on ZK for leader election.
Have each NN HA pair (cluster for that matter) talk to their own ZK quorum within the same network segment.