10-13-2016 05:41 AM
While running a large INSERT IGNORE INTO query which selects data from a Kudu table and reinserts this after some modification into another Kudu table.
While this query is running we see our logs fill up with a lot of the following error messages:
T c0f417f06cf4423cba426b9d4dbf95eb P 7390d2dddae2449db5547c8d4efbd839 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 900 is ahead of the local log (next sequential op: 900). Destination peer: Peer: 801c9bc30b2d4551a3833a5821532c31, Is new: false, Last received: 291.900, Next index: 901, Last known committed idx: 900, Last exchange result: ERROR, Needs tablet copy: false
We are unsure where this error is coming from and what the impact is. We are running Kudu 1.0.0-1.kudu1.0.0.p0.6 with Impala_Kudu 2.7.0-1.cdh5.9.0.p0.23 and CDH 5.8.0-1.cdh5.8.0.p0.42.
Can anyone point us in the direction of solving this?
10-13-2016 12:08 PM - edited 10-13-2016 12:09 PM
This seems an instance of https://issues.apache.org/jira/browse/KUDU-1078.
Our previous thinking was that this was a transient error that would eventually sort itself, though your particular use case of reading and writing at the same time might make things worse.
A few questions:
Is this actually causing the write to fail? What error do you see on the client side?
What is the cluster size and rough specs?
How big are these reads/writes, e.g. are you rewriting the whole output table each time? How big is it?
How many disks do you have? are they flash? did you specify a different disk for the WAL (i.e. did you pass the --fs_wal_dir flag)?