Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Maximum Hive Table Partitions allowed & recommended

avatar
Expert Contributor

What is the maximum number of partitions allowed for a Hive table? E.g. 2k ... 10k?

Are there any performance implications we should consider as we get close to this number?

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
7 REPLIES 7

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Mentor

@Andrew Grande Thanks for sharing the HBASE approach. Nice!!!

avatar
Master Mentor

What database are you using for Metastore? @Wes Floyd

avatar
Super Collaborator

When working with a table of 1000 partitions and having the Hive concurrency enabled, I once ran into some problems. I don't know if it is still an issue (the problem appeared last year with Hive 0.13) but I think it can be worth mentioning it here:

http://mail-archives.apache.org/mod_mbox/hive-user/201408.mbox/%3CCAENxBwxmjN7VTJuzq1G4FimoFYkwZsWJJ...

avatar
Contributor

The performance implications mostly come at read time. If you have queries that read many (>2k) partitions you will see long (30+ sec) times to plan queries. As Andrew mentioned, the work on the HBase metastore should improve this.

avatar
Master Mentor

Thanks @gates@hortonworks.com for chimmig in .

avatar
Rising Star

what if I only open less than 50 partitions out of 1M at any given time??