Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Solved Go to solution

WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

What does the warning mean?

 

I have a table having two level partitions like yearmonth and name_prefix.

Every month, I add new data with new yearmonth partition.

Once ETL is finished, I run `COMPUTE INCREMENTAL STATS`.

It takes about 2hr and it seems to take longer and longer.

 

Based on what I read, `COMPUTE INCREMENTAL STATS` only need to gather statistics for the partitions having false value for `Incremental stats`. 

 

However, the time I spend for running `COMPUTE STATS` and `COMPUTE INCREMENTAL STATS` is very similar.

 

Am I missing anything?

 

Thank you

Gatsby

 

+---------------------------------------------+
| summary                                     |
+---------------------------------------------+
| Updated 1372 partition(s) and 29 column(s). |
+---------------------------------------------+
WARNINGS: Too many partitions selected, doing full recomputation of incremental stats
Fetched 1 row(s) in 5871.22s

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Master Collaborator
3 REPLIES 3
Highlighted

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Master Collaborator

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

Re: WARNINGS: Too many partitions selected, doing full recomputation of incremental stats

Contributor

FYI, `COMPUTE STATS` can run with first level partition.

https://issues.cloudera.org/browse/IMPALA-1570

Don't have an account?
Coming from Hortonworks? Activate your account here