I resolved my problem. One transaction was still opened and all delta files with a transaction number upper wasn't processed. The code snippet which manages this case : /**
* Transform a {@link org.apache.hadoop.hive.metastore.api.GetOpenTxnsInfoResponse} to a
* {@link org.apache.hadoop.hive.common.ValidTxnList}. This assumes that the caller intends to
* compact the files, and thus treats only open transactions as invalid. Additionally any
* txnId > highestOpenTxnId is also invalid. This is avoid creating something like
* delta_17_120 where txnId 80, for example, is still open.
* @param txns txn list from the metastore
* @return a valid txn list.
*/
public static ValidTxnList createValidCompactTxnList(GetOpenTxnsInfoResponse txns) {
//todo: this could be more efficient: using select min(txn_id) from TXNS where txn_state=" +
// quoteChar(TXN_OPEN) to compute compute HWM...
long highWater = txns.getTxn_high_water_mark();
long minOpenTxn = Long.MAX_VALUE;
long[] exceptions = new long[txns.getOpen_txnsSize()];
int i = 0;
for (TxnInfo txn : txns.getOpen_txns()) {
if (txn.getState() == TxnState.OPEN) minOpenTxn = Math.min(minOpenTxn, txn.getId());
exceptions[i++] = txn.getId();//todo: only add Aborted
}//remove all exceptions < minOpenTxn
highWater = minOpenTxn == Long.MAX_VALUE ? highWater : minOpenTxn - 1;
return new ValidCompactorTxnList(exceptions, -1, highWater);
}
... View more
