question Re: Maximum Hive Table Partitions allowed & recommended in Archives of Support Questions (Read Only)

Maximum Hive Table Partitions allowed & recommended

wfloyd — Fri, 30 Oct 2015 21:42:18 GMT

What is the maximum number of partitions allowed for a Hive table? E.g. 2k ... 10k?

Are there any performance implications we should consider as we get close to this number?

Re: Maximum Hive Table Partitions allowed & recommended

andrewg — Fri, 30 Oct 2015 21:46:59 GMT

Wes, current Hive versions with RDBMS metastore backend should be able to handle 10 000+ partitions.

For numerous reasons, the community is moving away from this design to leverage HBase for the metastore. Follow https://issues.apache.org/jira/browse/HIVE-9452 . Overall design document is available here: https://issues.apache.org/jira/secure/attachment/12697601/HBaseMetastoreApproach.pdf

Re: Maximum Hive Table Partitions allowed & recommended

nsabharwal — Sat, 31 Oct 2015 00:53:22 GMT

What database are you using for Metastore? @Wes Floyd

Re: Maximum Hive Table Partitions allowed & recommended

nsabharwal — Sat, 31 Oct 2015 04:37:44 GMT

@Andrew Grande Thanks for sharing the HBASE approach. Nice!!!

Re: Maximum Hive Table Partitions allowed & recommended

sluangsay — Mon, 02 Nov 2015 16:16:53 GMT

When working with a table of 1000 partitions and having the Hive concurrency enabled, I once ran into some problems. I don't know if it is still an issue (the problem appeared last year with Hive 0.13) but I think it can be worth mentioning it here:

http://mail-archives.apache.org/mod_mbox/hive-user/201408.mbox/%3CCAENxBwxmjN7VTJuzq1G4FimoFYkwZsWJJW4xfiq-6uR+Hb7=cA@mail.gmail.com%3E

Re: Maximum Hive Table Partitions allowed & recommended

gates — Tue, 10 Nov 2015 01:03:11 GMT

The performance implications mostly come at read time. If you have queries that read many (>2k) partitions you will see long (30+ sec) times to plan queries. As Andrew mentioned, the work on the HBase metastore should improve this.

Re: Maximum Hive Table Partitions allowed & recommended

nsabharwal — Tue, 10 Nov 2015 02:18:48 GMT

Thanks @gates@hortonworks.com for chimmig in .

Re: Maximum Hive Table Partitions allowed & recommended

thewayofthinkin — Tue, 15 Mar 2016 04:44:03 GMT

what if I only open less than 50 partitions out of 1M at any given time??