06-09-2016 02:52 PM
I'm running an 'alter table add partition' to an external impala table, and it runs progressively slower each time.
I add 4k partitions per day to the external table. The dataset is about 22 million records in total.
It takes about 17 minutes to add all 4k partitions initially. Second run of adding 4k partitions take 40 min. The partition adds will progressively get slower until adding the partitions take several hours.
Can someone shed some light on what might be happening?