Reply
Explorer
Posts: 20
Registered: ‎09-13-2017

Re: Data replication in Kudu

Hi adar,

 

I think I already get the answer from your post. Maybe the following explanation can clarify my original question.

 

When I insert data into kudu,  only write to a majority's write-ahead logs. The internal flushing and/or compacting for each tablet will generate a set of CFiles as replicas.

 

And all scan only need to scan the replica (a set of CFiles which contain the base data and the delta data)  and the MemRowset to return the query result. Is this right?

 

But for the tablet coping, it will only copy the wal or both the wal and the replica will be copied?

 

Best regards,

Tony

 

Cloudera Employee
Posts: 19
Registered: ‎11-12-2013

Re: Data replication in Kudu

Quick note: Kudu calls the "set of CFiles which contain the base data and the delta data" a DiskRowSet.

 

But your understanding is correct: during a scan, the contents of the MemRowSet and some DiskRowSets are scanned for data. During a tablet copy, both the WAL segments and the CFiles are copied.

Explorer
Posts: 20
Registered: ‎09-13-2017

Re: Data replication in Kudu

Hi adar,

Thanks for your quick reply.

Best regards,
Tony
New Contributor
Posts: 3
Registered: ‎11-25-2018

Re: Data replication in Kudu

what is CFile and wal file ?
Expert Contributor
Posts: 96
Registered: ‎10-04-2017

Re: Data replication in Kudu

CFile is an on-disk columnar storage format which holds data and associated B-Tree indexes. 

 

https://github.com/cloudera/kudu/blob/master/docs/design-docs/cfile.md

Highlighted
Cloudera Employee
Posts: 66
Registered: ‎04-08-2014

Re: Data replication in Kudu

[ Edited ]

A WAL file is a Kudu tablet write-ahead log file. You can read an overview of how the Kudu write path works here (it's a fairly techincal blog post): https://blog.cloudera.com/blog/2017/04/apache-kudu-read-write-paths/

 

The WAL file location is controlled by the configuration parameter --fs_wal_dir which you can read about at https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_fs_wal_dir

Announcements