Support Questions

Find answers, ask questions, and share your expertise

Mutliple Hive Metastores.

New Contributor

I want to be able to split a single cluster into QA and DEV. File system wise I am happy how to do this but is it possible to run more than 1 Hive metastore (not for HA reasons) on the same cluster? The reason for wanting to do this is ensure that the QA and DEV Hive namespaces are separate and that they can refer to the same Hive database / table names.

4 REPLIES 4

Expert Contributor

@Maurice Hickey With HDP 3.x this will be much easier, but AFAIK this isn't as elegant prior to that. Keep in mind that there is one RDBMS maintaining the Hive Metadata - and therefore you can have only one Database/Table entry. In theory, you could have 2 x HS2/Metastores (QA/DEV) pointing to a different Metastore RDBMS each - but you lose the ability to query between the two (as DEV and QA are separate) unless you federate at a level above the cluster.

A common approach would be to split at the database level, so you can keep the same table names. You need to change for PROD, so would advise changing DB name is part of the promotion process to PROD anyway, so should be done from DEV to QA).

New Contributor

Hi Graham

Many thanks for your response. Unfortunately, the Hive database names are determined by the third party application we are using which is Podium (podiumdata.com). The database name is an integral part of their data structure so if difficult (read that as impossible) to have different database names between our dev / qa / prod areas. I do not need the ability to query between the two.

What I conceptually had in mind was indeed your suggestion of "In theory, you could have 2 x HS2/Metastores (QA/DEV) pointing to a different Metastore RDBMS each" but have been unable to find any direction on achieving such a setup.

Have you seen anything that would cast some light on the process of setting that up?

Out of interest, you referred to it being "easier with HDP 3.x", I had a look through the roadmap, what 3.x feature is coming up in this area?.

Master Collaborator

Running 2 Metastore/HS2 with different config is possible. If your cluster is Ambari managed, you could potentially achieve that by using Ambari Config Groups.

New Contributor

Thank you Deepesh, I will look into that.