question Re: HDFS Federation understanding in Archives of Support Questions (Read Only)

HDFS Federation understanding

AlinaGHERMAN — Sun, 08 Nov 2015 15:03:41 GMT

Hello,

I just tring to understand better the HDFS Federation.

If I get it right:

- we should use it in order to split for example the real time space and the batch space.

- if we want to split the namespace into N namespaces than we have to have N namenodes

Thank you!

Alina

Re: HDFS Federation understanding

Harsh J — Sun, 08 Nov 2015 15:27:57 GMT

Your understanding seems right, but note that none of the 'splitting' is automatic.

At its simplest form, federation is a way to have multiple distinct NameNodes powered by a common set of DataNodes.

Effectively, its running and managing 2 or more *separate* namespaces on top of the same storage space.

If you deploy two federated NameNodes, say hdfs://host-nn1/ and hdfs://host-nn2, then they will have nothing in common except the Live DN hostnames they share. A 'hadoop fs -ls' done on each will return absolutely independent results.