Support Questions
Find answers, ask questions, and share your expertise

Impala forgets about partitions between drop partition and insert shortly after

Explorer

Impala forgets about partitions between drop partition and insert shortly after:


We run a lot of:

 

ALTER TABLE reports.long_term DROP PARTITION (datepartition = '2000020106')

 


Before we are running a single large insert containing multiple partitions (the ones that just have been dropped):

 

 

insert into reports.long_term (
[...]
datepartition, -- partition key
reporttype -- partition key
)
values
(
[...]
"2000020106",
"report"
), [...]

 

 

The error message we get:

 

 

Updating metastore with 69 altered partitions (datepartition=2000020101/reporttype=report/ [...] , datepartition=2000020106/reporttype=report/, datep [...]
[...]
CAUSED BY: IllegalStateException: Invalid partition name: datepartition=2000020106/reporttype=report

 

 

 

So it wants to update the metastore which works fine for most partitions but not for some and in that list of partitions being updated - one suddenly fails with "invalid partition name".

 

As a workaround we had to adjust our insert statements grouped by partition keys which sadly makes our inserts slower as they are multiple queries now.

 

Has anyone encountered that issue before?

 

0 REPLIES 0