Data replication in Kudu

[#1]

Hi,

How is data replicated in Kudu? My understanding is that kudu has one replica of all data and 2 replicas with operational logs. From the apache docs i get this "Kudu does not replicate the on-disk storage of a tablet,
but rather just its operation log. The physical storage of each replica of a tablet is fully decoupled."

In case of a disk failure, if the disk contains the actual data and not operation log, how is it recovered??

[#2]

Hi Rajesh,

Right, Kudu replicates data logically to multiple tservers based on each table's replication factor (typically 3), and in doing so, writes are only considered successful once durably written to a majority's write-ahead logs. From then on, each tserver can maintain the data via flushing and compactions, "decoupled" from the writes to the log.

Currently, in cases of disk failures, the single failed node will crash. Because the data is written to at least a majority, Kudu "re-replicates" back up to full replication, i.e. all of the tablets that lost a replica because of the crash will notice that one of the servers is down and make a new copy on another healthy server.

[#3]

Hi Awong,

Right, Kudu replicates data logically to multiple tservers based on each table's replication factor (typically 3), and in doing so, writes are only considered successful once durably written to a majority's write-ahead logs. From then on, each tserver can maintain the data via flushing and compactions, "decoupled" from the writes to the log.

After the flushing and compaction of tserver, each tablet will have 2 physical replications. And the subsequent CLOSEST_REPLICA scan don't have to compact the wal, is this right?

Best regards,

Tony

[#4]

I'm not Andrew but I don't understand your question. Tablet replicas are flushed and/or compacted independently of one another, which means the physical layout of each one may be different; thus it would be incorrect to consider them "physically replicated".

Furthermore, all scans (whether CLOSEST_REPLICA or otherwise) are read-only operations and thus don't trigger WAL garbage collection or any other kind of read-write server-side action.

[#5]

Hi adar,

I think I already get the answer from your post. Maybe the following explanation can clarify my original question.

When I insert data into kudu,  only write to a majority's write-ahead logs. The internal flushing and/or compacting for each tablet will generate a set of CFiles as replicas.

And all scan only need to scan the replica (a set of CFiles which contain the base data and the delta data)  and the MemRowset to return the query result. Is this right?

But for the tablet coping, it will only copy the wal or both the wal and the replica will be copied?

Best regards,

Tony

[#6]

Quick note: Kudu calls the "set of CFiles which contain the base data and the delta data" a DiskRowSet.

But your understanding is correct: during a scan, the contents of the MemRowSet and some DiskRowSets are scanned for data. During a tablet copy, both the WAL segments and the CFiles are copied.

[#7]

Hi adar,

Thanks for your quick reply.

Best regards,
Tony

[#8]

what is CFile and wal file ?

[#9]

CFile is an on-disk columnar storage format which holds data and associated B-Tree indexes. 

https://github.com/cloudera/kudu/blob/master/docs/design-docs/cfile.md

[#10]

A WAL file is a Kudu tablet write-ahead log file. You can read an overview of how the Kudu write path works here (it's a fairly techincal blog post): https://blog.cloudera.com/blog/2017/04/apache-kudu-read-write-paths/

The WAL file location is controlled by the configuration parameter --fs_wal_dir which you can read about at https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_fs_wal_dir