Partition Spec Does Not Exist error

[#1]

Hi there,

We're seeing an intermittent issue with some of our Impala queries, where a compute incremental stats statement will fail with an "AnalysisException: Partition spec does not exist:" error, immediately after we have created the partition in the same script.  So, a (faked) example would look like:

ALTER TABLE db.table_name DROP IF EXISTS PARTITION (partition_column = 1234);

INSERT INTO TABLE 
   db.table_name 
PARTITION 
(
   partition_column = 1234
)
SELECT   
   column1
 , column2
 , ...
FROM
   db.another_table
;

COMPUTE INCREMENTAL STATS db.table_name (partition_column = 1234);

AnalysisException: Partition spec does not exist: (partition_column = 1234).

The tables in question are not being modifed by any external processes so we're not sure what could be happening here - has anyone else encountered this issue?

Thanks,

Mike

[#2 · Accepted solution]

This is a known issue, but I havent yet seen it without multiple concurent Impala writers.

(I have been able to repro it both in the same impalad and in different impalads.)

https://issues.cloudera.org/browse/IMPALA-2423

Is there only a single writer using a single impalad at the time?

[#3]

Thanks for the response, and  yes, is looks like that's the issue - going over our logs it sems that this only occurs when we have multiple impalads targeting different partitions in the same table.

Incidentally, I think we've seen this with as few as two concurrent operations, but in this case the two operations are likely to both start and complete at almost exactly the same times, so I guess there may be some race condition when updating the partition metadata?

[#4]

Could be. We're still looking into it.

[#5]

Hi jbapple,

I'm a colleague of Mike that originally submitted this. we're still hitting the issue  (Impala v2.3.0-cdh5.5.1), any update on whether or not this might be fixed?

regards

Jamie

[#6]

You can follow the bug I linked to above for any future updates. Right nowI have no news for you.

[#7]

---

@jbapple wrote:

You can follow the bug I linked to above for any future updates. Right nowI have no news for you.

---

DOH! Didn't even see that link. Must pay more attention, my apologies.