question WARNINGS: Too many partitions selected, doing full recomputation of incremental stats in Archives of Support Questions (Read Only)

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

thewayofthinkin — Thu, 31 Dec 2015 18:50:37 GMT

What does the warning mean?

I have a table having two level partitions like yearmonth and name_prefix.

Every month, I add new data with new yearmonth partition.

Once ETL is finished, I run `COMPUTE INCREMENTAL STATS`.

It takes about 2hr and it seems to take longer and longer.

Based on what I read, `COMPUTE INCREMENTAL STATS` only need to gather statistics for the partitions having false value for `Incremental stats`.

However, the time I spend for running `COMPUTE STATS` and `COMPUTE INCREMENTAL STATS` is very similar.

Am I missing anything?

Thank you

Gatsby

+---------------------------------------------+
| summary                                     |
+---------------------------------------------+
| Updated 1372 partition(s) and 29 column(s). |
+---------------------------------------------+
WARNINGS: Too many partitions selected, doing full recomputation of incremental stats
Fetched 1 row(s) in 5871.22s

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

alex.behm — Tue, 05 Jan 2016 00:47:12 GMT

See my response in the other thread:

http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/why-show-column-stats-lt-table-name-gt-doesn-t-show-statistics/m-p/35701#M1355

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

thewayofthinkin — Tue, 05 Jan 2016 01:18:16 GMT

following with the https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/why-show-column-stats-lt-table-name-gt-doesn-t-show-statistics/m-p/35681#U35681 post.

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

thewayofthinkin — Wed, 01 Feb 2017 00:36:03 GMT

FYI, `COMPUTE STATS` can run with first level partition.

https://issues.cloudera.org/browse/IMPALA-1570