Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data replication in Kudu

Solved Go to solution
Highlighted

Re: Data replication in Kudu

Contributor

Hi,

 

You can have multiple replicas of data stored in Kudu tables -- Kudu allows you to configure per-table replication factor when creating a table.  Replication factors of 3, 5, and 7 are available out of the box; for higher if you need to tweak the --max_num_replicas mater's flag.

 

Under the hood, every tablet (part of the table which corresponds to a partition) is a Raft cluster, where every transaction is considered committed only when it's replicated and acknowledged back to the leader replica by the majority of replicas in the tablet.

 

Replicas of one tablet are distributed among different tablet servers (it's not possible to run multiple replicas of one tablet at the same tablet server).  Unless the replication factor is set to 1 (i.e. no replication at all) or all tablet servers are run on the same machine (which is a bad idea), then for every tablet there should be at least one replica having the copy of the data once a disk on one server fails.

 

You can get more details at https://kudu.apache.org/overview.html#distribution-and-fault-tolerance

and https://github.com/apache/kudu/blob/master/docs/design-docs/consensus.md

 

I hope this helps. 

Re: Data replication in Kudu

Expert Contributor

Hi @Alexey1c @awong

 

Thanks for the information but sorry i couldn't understand. Do you mean we have different replication policies for tables and tablets??

Re: Data replication in Kudu

Contributor
Right, the replication factor is specified per table, so you could have
different tables with different replication factors. Every tablet in a
table will honor its table's replication factor.

Re: Data replication in Kudu

Expert Contributor
Hi @awong

Thanks for the details. So below is the conclusion.

Table: has replication specified at creation and all are complete replicas.

Tablet: has only operational log.

Re: Data replication in Kudu

Contributor

Somewhat. Take a look here for more details about the relationship between tablets and tablets: https://kudu.apache.org/docs/schema_design.html

 

There's an important distinction to be made: a tablet is a logical concept (it's a chunk of a table); a replica is a copy of a single tablet. There may be many replicas of a single tablet, depending on the user-specified properties of the table.

 

E.g. say I have "Table 1" with replication factor 3. This means that every tablet belonging to "Table 1" will always try to maintain 3 replicas/copies. Say "Table 1" has two tablets, "A" and "B", each will have three replicas. A replica of "A" could fail due to a server failure or somesuch, in which case "A" will try to replicate back up to having 3 healthy replicas. This is completely orthogonal to "B".

 

So yes, a tablet maintains its operational log, but also all of the data associated with it, because it is just a chunk of a table.

 

Hope this helped!

Re: Data replication in Kudu

Expert Contributor
@awong

That was great. Thanks for the details.