Reply
Explorer
Posts: 7
Registered: ‎08-19-2014

Read ahead of logs error while running large insert query

Hi,

 

While running a large INSERT IGNORE INTO query which selects data from a Kudu table and reinserts this after some modification into another Kudu table.


While this query is running we see our logs fill up with a lot of the following error messages:

 

T c0f417f06cf4423cba426b9d4dbf95eb P 7390d2dddae2449db5547c8d4efbd839 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 900 is ahead of the local log (next sequential op: 900). Destination peer: Peer: 801c9bc30b2d4551a3833a5821532c31, Is new: false, Last received: 291.900, Next index: 901, Last known committed idx: 900, Last exchange result: ERROR, Needs tablet copy: false

 

We are unsure where this error is coming from and what the impact is. We are running Kudu 1.0.0-1.kudu1.0.0.p0.6 with Impala_Kudu 2.7.0-1.cdh5.9.0.p0.23 and CDH 5.8.0-1.cdh5.8.0.p0.42. 

 

Can anyone point us in the direction of solving this?

Thank you,


Vincent 

Highlighted
Cloudera Employee
Posts: 4
Registered: ‎10-30-2015

Re: Read ahead of logs error while running large insert query

[ Edited ]

Hi Vincent

 

  This seems an instance of https://issues.apache.org/jira/browse/KUDU-1078.

  Our previous thinking was that this was a transient error that would eventually sort itself, though your particular use case of reading and writing at the same time might make things worse.

 

  A few questions:

  

  Is this actually causing the write to fail? What error do you see on the client side?

  What is the cluster size and rough specs?

  How big are these reads/writes, e.g. are you rewriting the whole output table each time? How big is it?

  How many disks do you have? are they flash? did you specify a different disk for the WAL (i.e. did you pass the --fs_wal_dir flag)?

 

Best

David

  

 

Announcements