question Impala/Kudu non-covering range partition to support rolling window data retention in Archives of Support Questions (Read Only)

Impala/Kudu non-covering range partition to support rolling window data retention

brianwhu — Fri, 16 Sep 2022 11:04:18 GMT

Hi Impala/Kudu gurus,

I'm extremely excited by the new Impala/Kudu release that supports non-covering range partition, as described here: https://github.com/cloudera/kudu/blob/master/docs/design-docs/non-covering-range-partitions.md

and here: https://gerrit.cloudera.org/#/c/4856/

Yet I haven't figured out how exactly to use it to support rolling window data retention that our business needs. The syntax descibed in the 2nd document above still seems to require static partition specification.

What we need is the ability to auto-create new partitions based on a timestamp expression so that each partition contains x days of data only. We then can drop the old partitions based on our data retention policy on a per table basis.

As a comparison, the similar function is provided by Oracle's range interval partition:

PARTITION BY RANGE (CREATION_DATE)
INTERVAL (NUMTODSINTERVAL(7, 'DAY'))

and Vertica's partition key expression:

PARTITION BY (floor((((tbl.creation_ts)::date - '0001-12-31 BC'::date) / 3)))

Thanks,

Brian

Re: Impala/Kudu non-covering range partition to support rolling window data retention

Matt Jacobs — Wed, 22 Mar 2017 18:11:43 GMT

Hi Brian,

Unfortunately Kudu partitions must be pre-defined as you suspected, so the Oracle syntax you described won't work for Impala. However, you can add and drop range partitions even after the table is created, so you can manually add the next hour/day/week partition, and drop some historical partition. The syntax is described in the latest version of the CDH documentation:

https://www.cloudera.com/documentation/enterprise/latest/topics/impala_kudu.html#kudu_range_partitioning

Best,

Matt

Re: Impala/Kudu non-covering range partition to support rolling window data retention

brianwhu — Thu, 13 Apr 2017 13:36:28 GMT

Hi Matt,

It seems we are going down this path for now. It is close enough to what we have in Vertica, which does grow new partitions automatically though.

Thanks,

Brian