Support Questions
Find answers, ask questions, and share your expertise

Impala table having large of number of partitions

Explorer

Hi All,

I have a table which is now having more than 100k partitions. The number of partitions is growing and has drastically degraded the performance. Just wanted to know meanwhile I try and understand how it can be partitioned in other ways, is there any parameters in catalog or metastore configs which can be tweaked to handle this scenario?

2 REPLIES 2

Rising Star

More that 100K partitions is not recommended with Impala.

 

The Impala cookbook recommends some techniques to reduce the number..

https://blog.cloudera.com/latest-impala-cookbook/

Rising Star

@kundansonuj 

 

Impala tables should have less than 30K partitions to ensure you get the necessary performances. The workaround would be to recreate tables with less partitions.

; ;