question Re: Partition Spec Does Not Exist error in Archives of Support Questions (Read Only)

Partition Spec Does Not Exist error

MikeA — Fri, 16 Sep 2022 09:46:15 GMT

Hi there,

We're seeing an intermittent issue with some of our Impala queries, where a compute incremental stats statement will fail with an "AnalysisException: Partition spec does not exist:" error, immediately after we have created the partition in the same script. So, a (faked) example would look like:

ALTER TABLE db.table_name DROP IF EXISTS PARTITION (partition_column = 1234);

INSERT INTO TABLE 
   db.table_name 
PARTITION 
(
   partition_column = 1234
)
SELECT   
   column1
 , column2
 , ...
FROM
   db.another_table
;

COMPUTE INCREMENTAL STATS db.table_name (partition_column = 1234);

AnalysisException: Partition spec does not exist: (partition_column = 1234).

The tables in question are not being modifed by any external processes so we're not sure what could be happening here - has anyone else encountered this issue?

Thanks,

Mike

Re: Partition Spec Does Not Exist error

jbapple — Tue, 27 Oct 2015 15:52:44 GMT

This is a known issue, but I havent yet seen it without multiple concurent Impala writers.

(I have been able to repro it both in the same impalad and in different impalads.)

https://issues.cloudera.org/browse/IMPALA-2423

Is there only a single writer using a single impalad at the time?

Re: Partition Spec Does Not Exist error

MikeA — Tue, 27 Oct 2015 17:53:03 GMT

Thanks for the response, and yes, is looks like that's the issue - going over our logs it sems that this only occurs when we have multiple impalads targeting different partitions in the same table.

Incidentally, I think we've seen this with as few as two concurrent operations, but in this case the two operations are likely to both start and complete at almost exactly the same times, so I guess there may be some race condition when updating the partition metadata?

Re: Partition Spec Does Not Exist error

jbapple — Tue, 27 Oct 2015 22:03:42 GMT

Could be. We're still looking into it.

Re: Partition Spec Does Not Exist error

jamiet — Mon, 20 Jun 2016 12:49:49 GMT

Hi jbapple,

I'm a colleague of Mike that originally submitted this. we're still hitting the issue (Impala v2.3.0-cdh5.5.1), any update on whether or not this might be fixed?

regards

Jamie

Re: Partition Spec Does Not Exist error

jbapple — Mon, 20 Jun 2016 13:35:44 GMT

You can follow the bug I linked to above for any future updates. Right nowI have no news for you.

Re: Partition Spec Does Not Exist error

jamiet — Mon, 20 Jun 2016 13:37:11 GMT

@jbapple wrote:
You can follow the bug I linked to above for any future updates. Right nowI have no news for you.

DOH! Didn't even see that link. Must pay more attention, my apologies.