COMPUTE INCREMENTAL STATS with PARTITION option: do I need to run DROP INCREMENTAL STATS before it?

@Me_01 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDP experts @PabitraDas and @aakulov  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Hi,

This statement in the doc

"In cases where new files are added to an existing partition, issue a REFRESH statement for the table, followed by a DROP INCREMENTAL STATS and COMPUTE INCREMENTAL STATS sequence for the changed partition."

Applies specifically to a partition in which stats are already available but you added more data to that existing partition.

If you are unsure about whether stats exist for a partition you can run show table stats <table_name>; and check the "Incremental stats" section 

Query: show table stats test_part
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------------------+
| b | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------------------+
| 1 | 0 | 1 | 0B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://xxxx:8020/user/hive/warehouse/test_part/b=1 |
| Total | -1 | 1 | 0B | 0B | | | | |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------------------------+
Fetched 2 row(s) in 5.60s

If false, you can run COMPUTE INCREMENTAL STATS with PARTITION 

If true and you have added more data to this partition then you have to drop the stats and then run COMPUTE INCREMENTAL STATS with PARTITION 

Hello, 

this is not answering my question. I'd like to know why I must run the DROP INCREMENTAL STATS in my scenario.

I have run a small test:

- created a partitioned table

- run COMPUTE INCREMENTAL STATS

- run SHOW TABLE STATS and saved the result

- added one row to one partition

- run COMPUTE INCREMENTAL STATS PARTITION (<partitioning key of the partition to which the row was added)

- run SHOW TABLE STATS and compare to the previous run

Result of the comparison: the stats of the partition to which I have added one row were updated and reflecting the number of rows in the partition / number of files.

Based on my tests it looks like the stats are updated, hence my question.

@Me_01 Sorry for that confusion. I see what you mean now

Per: https://impala.apache.org/docs/build/html/topics/impala_perf_stats.html#perf_stats_incremental

COMPUTE INCREMENTAL STATS

@Me_01 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. If you are still experiencing the issue, can you provide the information @venkatsambath has requested? Thanks.