Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

SOLVED Go to solution
Highlighted

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

What does the warning mean?

 

I have a table having two level partitions like yearmonth and name_prefix.

Every month, I add new data with new yearmonth partition.

Once ETL is finished, I run `COMPUTE INCREMENTAL STATS`.

It takes about 2hr and it seems to take longer and longer.

 

Based on what I read, `COMPUTE INCREMENTAL STATS` only need to gather statistics for the partitions having false value for `Incremental stats`. 

 

However, the time I spend for running `COMPUTE STATS` and `COMPUTE INCREMENTAL STATS` is very similar.

 

Am I missing anything?

 

Thank you

Gatsby

 

+---------------------------------------------+
| summary                                     |
+---------------------------------------------+
| Updated 1372 partition(s) and 29 column(s). |
+---------------------------------------------+
WARNINGS: Too many partitions selected, doing full recomputation of incremental stats
Fetched 1 row(s) in 5871.22s

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Master Collaborator
3 REPLIES 3

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Master Collaborator

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

FYI, `COMPUTE STATS` can run with first level partition.

https://issues.cloudera.org/browse/IMPALA-1570