Max number of databases and tables allowed in Hive Metastore

[#1]

Crossposting from: Stack Overflow

(I will respond and give updates on both places)

I would like to know if there is also maximum number for items below:

1. Maximum number of databases in a catalog (I assume Hive Metastore only has one catalog, which is "hive")
2. Maximum number of tables per database (as in, can I create 10 million tables in a database or due to limitation must I split them into 10 databases each with 1 million tables)

I also would like to know whether the limitations are hard limit (unconfigurable), or configurable by Hive, or dependent on RDBMS it is using.

[#2 · Accepted solution]

Hi,

There's no limit about the number of databases in Hive metastore. As a good practice we do not recommend to create tables with more than 10,0000 partitions. 

In my opinion, you shouldn't have problem with 2,000 tables. You can expect to have some type of performance issues if you have a total number object greater than 500,000 and 

there's no a hard limit about the number of Hive/Impala databases/tables  that you can have in the cluster.

Regards,

Chethan YM

[#3]

Thank you for the explanation. To clarify, what is the definition of "total number object"? Does it refer to total of "metadata objects", as defined in this page about Hive Design?

[#4]

Hi,

Yes, Hive metastore is a component that stores all the structure information(metadata) of objects like tables and partitions in the warehouse including column and column type information etc...

Regards,

Chethan YM

Note: If this answered your question please accept the reply as a solution.