Context
In Apache HBase replication, RegionServers in the source cluster contact RegionServers in the target cluster via RPC. When security is enabled, authentication is performed at the RPC connection establishment phase using the Simple Authentication and Security Layer framework (SASL). HBase already provides the following built-in SASL authentication mechanisms: Kerberos, digest and simple. When Kerberos is enabled, credentials from the source cluster will be expected by the target cluster, which will then validate these credentials against its own KDC, using the SASL Kerberos mechanism. This relies on Kerberos GSSAPI implementation for authenticating the provided credentials against the target cluster KDC, therefore the trust for the source cluster principal must have been implemented at the Kerberos system level, by either having both cluster's credentials on the same realm, or making the target cluster KDC trust the credentials from the source cluster realm (an approach commonly known as cross-realm authentication).
SASL is designed to allow for custom authentication implementations. Cloudera proposed a refactoring of HBase’s RPC layer, which has been reviewed and accepted by the Apache HBase community in HBASE-23347, making additional SASL mechanisms pluggable to the already existing built-in options mentioned above.
The Cloudera Replication Plugin is a custom, Cloudera only extension to the built-in HBase Replication framework first released in 2020. It implements an extra SASL authentication mechanism, based on Java encrypted certificate files, and uses it for the authentication of peer clusters, without requiring any additional Kerberos integration. It extends HBase built-in replication framework to use this additional SASL mechanism for authenticating inter-cluster RPC connections.
As of 2026, six years following the plugin’s first release, it has been a pillar for both data migration and disaster recovery (DR) use cases, and it has now been extended to support varying HBase releases as replication sources:
- CDH 5
- CDH 6
- HDP 2.6
- HDP 3.1
- EMR 5
- EMR 6
It has also expanded beyond the initial supported secured-to-secured or unsecured-to-secured cluster replication topologies to also accommodate secured-to-unsecured scenarios, as well as proxied endpoints for target clusters under private networks.
Extensions to the CopyTable and SyncTable utility tools are also available, enabling a complete HBase replication experience over the different security combinations mentioned above.
One missing piece left unsolved by the original Cloudera Replication Plugin design was ZooKeeper connections to the target cluster. Back when the initial plugin version was under development, information from the target cluster, like cluster ID and the list of its RegionServer addresses required by the replication source was only exposed in the target's ZooKeeper service. Due to the lack of support for similar pluggable authentication types in the ZooKeeper client library, we had to relax our security constraints when reading such info from the target ZooKeeper service. ZooKeeper release in use at the time didn't enforce authentication, meaning any unauthenticated client could still access information on its znodes. That is mitigated by the use of ACLs for the HBase znodes and since HBase only stores cluster related metadata (not user data) such as server addresses and region locations, that was a minor compromise.
Strengthening Security
Within the ever evolving open source ecosystem, both ZooKeeper and HBase have seen new functionalities being implemented and included in latest releases that allow for eliminating that last pitfall in ZooKeeper data protection. First, ZooKeeper incorporated the “authentication enforcement” feature (ZOOKEEPER-1634), which is configurable and once activated, it blocks non-authenticated clients from accessing znodes. The alternative was to modify the plugin, to access the list of RSes via the target Master RPC API, which already exposes this. But to establish this RPC connection, the client (the replication source) still needs to resolve the target cluster active master address and cluster ID. This is performed internally in the client RPC connection, and by default, it also relies on ZooKeeper data, which now isn’t available for the replication source because it lacks Kerberos credentials for the target cluster.
Another important contribution implemented by the Apache HBase community after the plugin's original release was HBASE-23305, which resolves master addresses via configuration and enables establishing RPC connections between clients and HBase servers without needing to fetch metadata from ZooKeeper. This was essential to allow the Cloudera Replication Plugin to connect to target clusters that have ZooKeeper Enforced Authentication enabled.
ZooKeeperless RPC Connections
With no means to access data from the target cluster’s ZooKeeper, the Cloudera Replication Plugin Endpoint needs to switch from the default ZooKeeper based connection registry to the new Master RPC only introduced in HBASE-23305 prior to creating a connection instance. This master RPC registry relies on local cluster HBase master configurations to resolve the master addresses, so the plugin needs to define an extra configuration property for specifying the target cluster's master addresses. It then uses this new property to override the configuration it passes for the connection to the target cluster.
Now every meta information from remote peers required by the source replication endpoint is fetched via RPC, using the same custom SASL mechanism originally implemented by the Cloudera Replication Plugin. Target clusters can enable enforced ZooKeeper Authentication to strengthen security, without impacting replication setups.
Conclusion
The Cloudera Replication Plugin has evolved to address the security constraints imposed by enforced ZooKeeper Authentication in modern HBase clusters. By leveraging the RPC connection registry introduced in HBASE-23305, the plugin can now successfully establish connections and fetch necessary metadata from target clusters solely via the Master RPC API. This critical enhancement allows users to enable ZooKeeper Enforced Authentication on their target clusters for strengthened security without disrupting existing replication flows, ensuring the plugin remains a robust and secure solution for inter-cluster data replication and disaster recovery use cases.
