question Data replication in Kudu in Support Questions

Data replication in Kudu

RajeshBodolla — Fri, 16 Sep 2022 12:31:15 GMT

Hi,

How is data replicated in Kudu? My understanding is that kudu has one replica of all data and 2 replicas with operational logs. From the apache docs i get this "Kudu does not replicate the on-disk storage of a tablet,
but rather just its operation log. The physical storage of each replica of a tablet is fully decoupled."

In case of a disk failure, if the disk contains the actual data and not operation log, how is it recovered??

Re: Data replication in Kudu

awong — Tue, 14 Nov 2017 18:32:27 GMT

Hi Rajesh,

Right, Kudu replicates data logically to multiple tservers based on each table's replication factor (typically 3), and in doing so, writes are only considered successful once durably written to a majority's write-ahead logs. From then on, each tserver can maintain the data via flushing and compactions, "decoupled" from the writes to the log.

Currently, in cases of disk failures, the single failed node will crash. Because the data is written to at least a majority, Kudu "re-replicates" back up to full replication, i.e. all of the tablets that lost a replica because of the crash will notice that one of the servers is down and make a new copy on another healthy server.

Re: Data replication in Kudu

Alexey1c — Tue, 14 Nov 2017 18:48:44 GMT

Hi,

You can have multiple replicas of data stored in Kudu tables -- Kudu allows you to configure per-table replication factor when creating a table. Replication factors of 3, 5, and 7 are available out of the box; for higher if you need to tweak the --max_num_replicas mater's flag.

Under the hood, every tablet (part of the table which corresponds to a partition) is a Raft cluster, where every transaction is considered committed only when it's replicated and acknowledged back to the leader replica by the majority of replicas in the tablet.

Replicas of one tablet are distributed among different tablet servers (it's not possible to run multiple replicas of one tablet at the same tablet server). Unless the replication factor is set to 1 (i.e. no replication at all) or all tablet servers are run on the same machine (which is a bad idea), then for every tablet there should be at least one replica having the copy of the data once a disk on one server fails.

You can get more details at https://kudu.apache.org/overview.html#distribution-and-fault-tolerance

and https://github.com/apache/kudu/blob/master/docs/design-docs/consensus.md

I hope this helps.

Re: Data replication in Kudu

RajeshBodolla — Wed, 15 Nov 2017 12:32:23 GMT

Hi @Alexey1c @awong

Thanks for the information but sorry i couldn't understand. Do you mean we have different replication policies for tables and tablets??

Re: Data replication in Kudu

awong — Wed, 15 Nov 2017 17:17:35 GMT

Right, the replication factor is specified per table, so you could have
different tables with different replication factors. Every tablet in a
table will honor its table's replication factor.

Re: Data replication in Kudu

RajeshBodolla — Wed, 15 Nov 2017 20:42:41 GMT

Hi @awong

Thanks for the details. So below is the conclusion.

Table: has replication specified at creation and all are complete replicas.

Tablet: has only operational log.

Re: Data replication in Kudu

awong — Wed, 15 Nov 2017 21:12:43 GMT

Somewhat. Take a look here for more details about the relationship between tablets and tablets: https://kudu.apache.org/docs/schema_design.html

There's an important distinction to be made: a tablet is a logical concept (it's a chunk of a table); a replica is a copy of a single tablet. There may be many replicas of a single tablet, depending on the user-specified properties of the table.

E.g. say I have "Table 1" with replication factor 3. This means that every tablet belonging to "Table 1" will always try to maintain 3 replicas/copies. Say "Table 1" has two tablets, "A" and "B", each will have three replicas. A replica of "A" could fail due to a server failure or somesuch, in which case "A" will try to replicate back up to having 3 healthy replicas. This is completely orthogonal to "B".

So yes, a tablet maintains its operational log, but also all of the data associated with it, because it is just a chunk of a table.

Hope this helped!

Re: Data replication in Kudu

RajeshBodolla — Wed, 15 Nov 2017 21:59:07 GMT

@awong

That was great. Thanks for the details.

Re: Data replication in Kudu

tony12 — Tue, 02 Jan 2018 07:20:12 GMT

Hi Awong,

Right, Kudu replicates data logically to multiple tservers based on each table's replication factor (typically 3), and in doing so, writes are only considered successful once durably written to a majority's write-ahead logs. From then on, each tserver can maintain the data via flushing and compactions, "decoupled" from the writes to the log.

After the flushing and compaction of tserver, each tablet will have 2 physical replications. And the subsequent CLOSEST_REPLICA scan don't have to compact the wal, is this right?

Best regards,

Tony

Re: Data replication in Kudu

adar — Tue, 02 Jan 2018 21:56:11 GMT

I'm not Andrew but I don't understand your question. Tablet replicas are flushed and/or compacted independently of one another, which means the physical layout of each one may be different; thus it would be incorrect to consider them "physically replicated".

Furthermore, all scans (whether CLOSEST_REPLICA or otherwise) are read-only operations and thus don't trigger WAL garbage collection or any other kind of read-write server-side action.

Re: Data replication in Kudu

tony12 — Wed, 03 Jan 2018 01:09:56 GMT

Hi adar,

I think I already get the answer from your post. Maybe the following explanation can clarify my original question.

When I insert data into kudu, only write to a majority's write-ahead logs. The internal flushing and/or compacting for each tablet will generate a set of CFiles as replicas.

And all scan only need to scan the replica (a set of CFiles which contain the base data and the delta data) and the MemRowset to return the query result. Is this right?

But for the tablet coping, it will only copy the wal or both the wal and the replica will be copied?

Best regards,

Tony

Re: Data replication in Kudu

adar — Wed, 03 Jan 2018 21:07:27 GMT

Quick note: Kudu calls the "set of CFiles which contain the base data and the delta data" a DiskRowSet.

But your understanding is correct: during a scan, the contents of the MemRowSet and some DiskRowSets are scanned for data. During a tablet copy, both the WAL segments and the CFiles are copied.

Re: Data replication in Kudu

tony12 — Thu, 04 Jan 2018 00:57:59 GMT

Hi adar,

Thanks for your quick reply.

Best regards,
Tony

Re: Data replication in Kudu

Ym255 — Sun, 25 Nov 2018 16:30:22 GMT

what is CFile and wal file ?

Re: Data replication in Kudu

RajeshBodolla — Tue, 27 Nov 2018 10:54:29 GMT

CFile is an on-disk columnar storage format which holds data and associated B-Tree indexes.

https://github.com/cloudera/kudu/blob/master/docs/design-docs/cfile.md

Re: Data replication in Kudu

mpercy — Tue, 27 Nov 2018 21:03:18 GMT

A WAL file is a Kudu tablet write-ahead log file. You can read an overview of how the Kudu write path works here (it's a fairly techincal blog post): https://blog.cloudera.com/blog/2017/04/apache-kudu-read-write-paths/

The WAL file location is controlled by the configuration parameter --fs_wal_dir which you can read about at https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_fs_wal_dir

Re: Data replication in Kudu

chn — Wed, 25 Aug 2021 08:07:22 GMT

hi,adar:

if both the WAL segments and the CFiles are copied duing a tablet copy,then the follower tablet will alse flushing wal data to disk when growing up to 8M,in my opinion there has no difference between master tablet and follower tablet during the reading and writing,is that right?