Skip to main content
felixnc
Silver ⭐️
Aug 29, 2022 8:30 AM UTC
Solved

Max number of databases and tables allowed in Hive Metastore

Crossposting from: Stack Overflow

(I will respond and give updates on both places)

I would like to know if there is also maximum number for items below:

  1. Maximum number of databases in a catalog (I assume Hive Metastore only has one catalog, which is "hive")
  2. Maximum number of tables per database (as in, can I create 10 million tables in a database or due to limitation must I split them into 10 databases each with 1 million tables)

I also would like to know whether the limitations are hard limit (unconfigurable), or configurable by Hive, or dependent on RDBMS it is using.

ChethanYM
ChethanYMAnswer
Gold ⭐️
Aug 29, 2022 10:43 AM UTC

Hi,

 

There's no limit about the number of databases in Hive metastore. As a good practice we do not recommend to create tables with more than 10,0000 partitions. 

 

In my opinion, you shouldn't have problem with 2,000 tables. You can expect to have some type of performance issues if you have a total number object greater than 500,000 and 

there's no a hard limit about the number of Hive/Impala databases/tables  that you can have in the cluster.

 

Regards,

Chethan YM

3 replies

ChethanYM
Gold ⭐️
ChethanYMAnswer
Gold ⭐️
Aug 29, 2022 10:43 AM UTC

Hi,

 

There's no limit about the number of databases in Hive metastore. As a good practice we do not recommend to create tables with more than 10,0000 partitions. 

 

In my opinion, you shouldn't have problem with 2,000 tables. You can expect to have some type of performance issues if you have a total number object greater than 500,000 and 

there's no a hard limit about the number of Hive/Impala databases/tables  that you can have in the cluster.

 

Regards,

Chethan YM

Translate
felixnc
felixncAuthor
Silver ⭐️
Aug 29, 2022 11:33 AM UTC

Thank you for the explanation. To clarify, what is the definition of "total number object"? Does it refer to total of "metadata objects", as defined in this page about Hive Design?

Translate
ChethanYM
Gold ⭐️
Gold ⭐️
Aug 29, 2022 12:07 PM UTC

Hi,

 

Yes, Hive metastore is a component that stores all the structure information(metadata) of objects like tables and partitions in the warehouse including column and column type information etc...

 

Regards,

Chethan YM

 

Note: If this answered your question please accept the reply as a solution.

Translate