Reply
Explorer
Posts: 6
Registered: ‎06-08-2016

Impala/KUDU Interface on achieving Write Quorum

Hi,

 
  I wanted to know if Impala and Kudu expose any interface to indicate achieving write quorum.
 
Ex: I do Impala Table inserts and before I read from the same table I want to know if the default replication of (3) has been achieved and any of my subsequent reads will not hit a node which is still waiting on eventual consistency.
For future I want to do the same on Kudu Tables, so does KUDU expose any such interface?
 
Any help is appreciated.
 
Thanks,
Pavan
Cloudera Employee
Posts: 65
Registered: ‎09-28-2015

Re: Impala/KUDU Interface on achieving Write Quorum

Hi Pavan,

 

 

First, I want to be clear on the semantics of Kudu's writes. We aren't "eventually consistent" in the way that the term is usually used when describing systems such as Cassandra or Riak. Kudu uses a strict consensus algorithm (Raft) which 100% guarantees that all replicas will perform the same write operations in the exact same order, and therefore always have identical timelines and result in identical database states. Of course, one replica (of three) may be lagging behind in the timeline at any point, but divergent writes are not possible.

 

When you see consistency aberrations in Impala, it's due to the fact that the Impala integration is currently configured with a loose _read_ consistency. In other words, Impala scanners may read from replicas which are lagging, and thus give unexpected results. This improves data locality (and thus throughput), because Impala has more flexibility of which replica it can read from for a given tablet. But, the consistency may not be satisfactory for some applications.

 

Given that, we plan to improve this in the future by having systems like Impala verify that the replica they are reading from is up-to-date to a given snapshot timestamp before performing the read. There are a number of small issues that need to be ironed out with this, which we will address in upcoming releases.


You can read more detail about Kudu's consistency model and known aberrations here: http://kudu.apache.org/docs/transaction_semantics.html

 

Hope that helps

-Todd

Explorer
Posts: 6
Registered: ‎06-08-2016

Re: Impala/KUDU Interface on achieving Write Quorum

Thanks a lot for the reply Todd. Looking forward to the future releases, would definitely help in delegating the check for up-to date read to the system itself.