Support Questions
Find answers, ask questions, and share your expertise

Federation and hbase

Explorer

Hi,

 

Because our most data will be in hbase, we want to use below folder structure for HDFS federation

/hbase/prj1
/hbase/prj2
..
/hbase/prjN

I would like to ask if it is ok for hbase to use its subfolders in different namespaces?

Thanks,

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Federation and hbase

Master Guru
Its easy to get carried away by the words "namespace separation" and what
it truly means in federation setup. Its not as simple as it sounds.

If you have 3 NNs, you will have 3 NNs with "/" each. Its not 3 NNs with
"/hbase" on all of them and files distributed automatically among them.

ViewFS is a client-sided FS that can talk to the 3 NNs, and rewrite the
path locally. That is, you can choose to mount / of NN1 as /hbase1, / of
NN2 as /hbase2 and / of NN3 as /hbase3, but not bind all of them to a
common root (/).

When you then ask for viewfs:///hbase1/foo, it internally goes to NN1
looking for just /foo. NN1 does not know what "/hbase1" is, only the client
(viewfs) does, which translates/brokers the request for you automatically.

Simply said, no - such a setup with a single HBase cluster over 3x NNs is
not possible.

Even if you did provide some form of viewfs:// as the root directory in
HBase config, it would fail cause of viewfs restrictions around moving
paths between mount-point paths.

View solution in original post

2 REPLIES 2

Re: Federation and hbase

Master Guru
Its easy to get carried away by the words "namespace separation" and what
it truly means in federation setup. Its not as simple as it sounds.

If you have 3 NNs, you will have 3 NNs with "/" each. Its not 3 NNs with
"/hbase" on all of them and files distributed automatically among them.

ViewFS is a client-sided FS that can talk to the 3 NNs, and rewrite the
path locally. That is, you can choose to mount / of NN1 as /hbase1, / of
NN2 as /hbase2 and / of NN3 as /hbase3, but not bind all of them to a
common root (/).

When you then ask for viewfs:///hbase1/foo, it internally goes to NN1
looking for just /foo. NN1 does not know what "/hbase1" is, only the client
(viewfs) does, which translates/brokers the request for you automatically.

Simply said, no - such a setup with a single HBase cluster over 3x NNs is
not possible.

Even if you did provide some form of viewfs:// as the root directory in
HBase config, it would fail cause of viewfs restrictions around moving
paths between mount-point paths.

View solution in original post

Re: Federation and hbase

Explorer

Thanks for the answer,

 

We are going to look increasing block size to 1GB+ to reduce the NN heap size. All data will reside in hbase, any other solutions other than seperating the cluster into physical partitions?