question Re: Federation and hbase in Archives of Support Questions (Read Only)

Federation and hbase

scobanx — Fri, 16 Sep 2022 10:02:11 GMT

Hi,

Because our most data will be in hbase, we want to use below folder structure for HDFS federation

/hbase/prj1
/hbase/prj2
..
/hbase/prjN

I would like to ask if it is ok for hbase to use its subfolders in different namespaces?

Thanks,

Re: Federation and hbase

Harsh J — Thu, 04 Feb 2016 09:30:52 GMT

Its easy to get carried away by the words "namespace separation" and what
it truly means in federation setup. Its not as simple as it sounds.

If you have 3 NNs, you will have 3 NNs with "/" each. Its not 3 NNs with
"/hbase" on all of them and files distributed automatically among them.

ViewFS is a client-sided FS that can talk to the 3 NNs, and rewrite the
path locally. That is, you can choose to mount / of NN1 as /hbase1, / of
NN2 as /hbase2 and / of NN3 as /hbase3, but not bind all of them to a
common root (/).

When you then ask for viewfs:///hbase1/foo, it internally goes to NN1
looking for just /foo. NN1 does not know what "/hbase1" is, only the client
(viewfs) does, which translates/brokers the request for you automatically.

Simply said, no - such a setup with a single HBase cluster over 3x NNs is
not possible.

Even if you did provide some form of viewfs:// as the root directory in
HBase config, it would fail cause of viewfs restrictions around moving
paths between mount-point paths.

Re: Federation and hbase

scobanx — Fri, 05 Feb 2016 08:10:50 GMT

Thanks for the answer,

We are going to look increasing block size to 1GB+ to reduce the NN heap size. All data will reside in hbase, any other solutions other than seperating the cluster into physical partitions?