Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala table having large of number of partitions

Impala table having large of number of partitions

Explorer

Hi All,

I have a table which is now having more than 100k partitions. The number of partitions is growing and has drastically degraded the performance. Just wanted to know meanwhile I try and understand how it can be partitioned in other ways, is there any parameters in catalog or metastore configs which can be tweaked to handle this scenario?

2 REPLIES 2
Highlighted

Re: Impala table having large of number of partitions

Contributor

More that 100K partitions is not recommended with Impala.

 

The Impala cookbook recommends some techniques to reduce the number..

https://blog.cloudera.com/latest-impala-cookbook/

Highlighted

Re: Impala table having large of number of partitions

Contributor

@kundansonuj 

 

Impala tables should have less than 30K partitions to ensure you get the necessary performances. The workaround would be to recreate tables with less partitions.

Don't have an account?
Coming from Hortonworks? Activate your account here