Reply
Highlighted
Contributor
Posts: 66
Registered: ‎12-30-2015
Accepted Solution

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

What does the warning mean?

 

I have a table having two level partitions like yearmonth and name_prefix.

Every month, I add new data with new yearmonth partition.

Once ETL is finished, I run `COMPUTE INCREMENTAL STATS`.

It takes about 2hr and it seems to take longer and longer.

 

Based on what I read, `COMPUTE INCREMENTAL STATS` only need to gather statistics for the partitions having false value for `Incremental stats`. 

 

However, the time I spend for running `COMPUTE STATS` and `COMPUTE INCREMENTAL STATS` is very similar.

 

Am I missing anything?

 

Thank you

Gatsby

 

+---------------------------------------------+
| summary                                     |
+---------------------------------------------+
| Updated 1372 partition(s) and 29 column(s). |
+---------------------------------------------+
WARNINGS: Too many partitions selected, doing full recomputation of incremental stats
Fetched 1 row(s) in 5871.22s

 

Cloudera Employee
Posts: 307
Registered: ‎10-16-2013

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor
Posts: 66
Registered: ‎12-30-2015

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor
Posts: 66
Registered: ‎12-30-2015

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

FYI, `COMPUTE STATS` can run with first level partition.

https://issues.cloudera.org/browse/IMPALA-1570