Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

avatar
Rising Star

What does the warning mean?

 

I have a table having two level partitions like yearmonth and name_prefix.

Every month, I add new data with new yearmonth partition.

Once ETL is finished, I run `COMPUTE INCREMENTAL STATS`.

It takes about 2hr and it seems to take longer and longer.

 

Based on what I read, `COMPUTE INCREMENTAL STATS` only need to gather statistics for the partitions having false value for `Incremental stats`. 

 

However, the time I spend for running `COMPUTE STATS` and `COMPUTE INCREMENTAL STATS` is very similar.

 

Am I missing anything?

 

Thank you

Gatsby

 

+---------------------------------------------+
| summary                                     |
+---------------------------------------------+
| Updated 1372 partition(s) and 29 column(s). |
+---------------------------------------------+
WARNINGS: Too many partitions selected, doing full recomputation of incremental stats
Fetched 1 row(s) in 5871.22s

 

1 ACCEPTED SOLUTION

avatar
3 REPLIES 3

avatar

avatar
Rising Star

avatar
Rising Star

FYI, `COMPUTE STATS` can run with first level partition.

https://issues.cloudera.org/browse/IMPALA-1570