- last edited on
Impala forgets about partitions between drop partition and insert shortly after:
We run a lot of:
ALTER TABLE reports.long_term DROP PARTITION (datepartition = '2000020106')
Before we are running a single large insert containing multiple partitions (the ones that just have been dropped):
insert into reports.long_term (
datepartition, -- partition key
reporttype -- partition key
The error message we get:
Updating metastore with 69 altered partitions (datepartition=2000020101/reporttype=report/ [...] , datepartition=2000020106/reporttype=report/, datep [...]
CAUSED BY: IllegalStateException: Invalid partition name: datepartition=2000020106/reporttype=report
So it wants to update the metastore which works fine for most partitions but not for some and in that list of partitions being updated - one suddenly fails with "invalid partition name".
As a workaround we had to adjust our insert statements grouped by partition keys which sadly makes our inserts slower as they are multiple queries now.
Has anyone encountered that issue before?