I'm trying to figure out how to use HDFS nameservices and/or ViewFS to simplify cross-cluster access in our data centers. Ideally I'd be able to produce a solution that allows clients to refer to data on different clusters by logical name, without needing to know any server names. Here's a typical example. There are two clusters, A and B. Neither is federated, and A is using a single namenode (call that host A-nn), while B is set up in a high availability configuration with two namenodes (call those hosts B-nn1 and B-nn2). Cluster A's configuration info is very plain:
A core-site.xml:
fs.defaultFS: hdfs://A-nn
Cluster B has nameservices set up with the following settings:
B core-site.xml:
fs.defaultFS: hdfs://B
B hdfs-site.xml:
dfs.internal.nameservices: B
dfs.nameservices: B
dfs.ha.namenodes.B: nn1,nn2
dfs.namenode.http-address.B.nn1: B-nn1:50070
dfs.namenode.https-address.B.nn1: B-nn1:50470
dfs.namenode.rpc-address.B.nn1: B-nn1:8020
dfs.namenode.http-address.B.nn2: B-nn2:50070
dfs.namenode.https-address.B.nn2: B-nn2:50470
dfs.namenode.rpc-address.B.nn2: B-nn2:8020
I'd like to get to a situation where clients could reference data on either cluster via URLs like hdfs://A/whatever or hdfs://B/whatever (or viewfs://A/whatever and viewfs://B/whatever if that's what it takes).
I've tried setting up cluster A as a pseudo-HA conf with only one namenode, like this:
A core-site.xml:
fs.defaultFS: hdfs://A
A hdfs-site.xml:
dfs.internal.nameservices: A
dfs.nameservices: A
dfs.ha.namenodes.A: nn1
dfs.namenode.http-address.A.nn1: A-nn1:50070
dfs.namenode.https-address.A.nn1: A-nn1:50470
dfs.namenode.rpc-address.A.nn1: A-nn1:8020
And that seems to work fine, allowing clients on cluster A to access data using hdfs://A/whatever URLs. Encouraged by that, I added the B info to A's config and the A info to B's config, so they wound up looking like this:
A core-site.xml:
fs.defaultFS: hdfs://A
A hdfs-site.xml:
dfs.internal.nameservices: A
dfs.nameservices: A,B
dfs.ha.namenodes.A: nn1
dfs.namenode.http-address.A.nn1: A-nn1:50070
dfs.namenode.https-address.A.nn1: A-nn1:50470
dfs.namenode.rpc-address.A.nn1: A-nn1:8020
dfs.ha.namenodes.B: nn1,nn2
dfs.namenode.http-address.B.nn1: B-nn1:50070
dfs.namenode.https-address.B.nn1: B-nn1:50470
dfs.namenode.rpc-address.B.nn1: B-nn1:8020
dfs.namenode.http-address.B.nn2: B-nn2:50070
dfs.namenode.https-address.B.nn2: B-nn2:50470
dfs.namenode.rpc-address.B.nn2: B-nn2:8020
B core-site.xml:
fs.defaultFS: hdfs://B
B hdfs-site.xml:
dfs.internal.nameservices: B
dfs.nameservices: A,B
dfs.ha.namenodes.A: nn1
dfs.namenode.http-address.A.nn1: A-nn1:50070
dfs.namenode.https-address.A.nn1: A-nn1:50470
dfs.namenode.rpc-address.A.nn1: A-nn1:8020
dfs.ha.namenodes.B: nn1,nn2
dfs.namenode.http-address.B.nn1: B-nn1:50070
dfs.namenode.https-address.B.nn1: B-nn1:50470
dfs.namenode.rpc-address.B.nn1: B-nn1:8020
dfs.namenode.http-address.B.nn2: B-nn2:50070
dfs.namenode.https-address.B.nn2: B-nn2:50470
dfs.namenode.rpc-address.B.nn2: B-nn2:8020
In other words, the configurations are now identical except for the fs.defaultFS and dfs.internal.nameservices settings. And this all appears to work fine (I'm doing this in alternate config directories and testing clients using the --config param, of course, so as not to affect normal operations).
However, reading the discussion in https://community.hortonworks.com/questions/71980/with-two-ha-clusters-configured-for-cross-cluster.... has me concerned that this may not be safe. I would prefer to distribute the same configuration files to all nodes in a cluster, but that article seems to suggest that the safest route is to distribute different versions to client nodes and non-client nodes. If necessary I can do that, but it will require additional work. Cluster A is managed via puppet, and we already have different roles for the client nodes and non-client nodes, so we can probably use that same mechanism to manage the configuration differences. Cluster B is managed via Ambari and I'm less sure what to do there. Presumably it will require different config groups and different settings in the custom hdfs-site settings for each config group?
We currently run our balancer processes from the client/gateway nodes, and we absolutely cannot afford to have the balancers inadvertently treat the separate clusters as a common block pool and distribute blocks across the clusters. If need be we can set up the balancers to run from non-client nodes, but if running the balancer from a client node is risky, that seems like a disaster waiting to happen; somebody will eventually do it by accident. Is limiting dfs.internal.nameservices to the local cluster really enough to prevent the balancer from treating the clusters named in dfs.nameservices as a single federated cluster?
Finally, it doesn't really seem like ViewFS brings much to this particular party. It seems to me that its only real value comes when using federated namenodes, which doesn't apply to our situation. Am I missing something there?
