Runing Compactor against multiple Hive Metastores?

[#1]

Hello Experts,

In Ambari 2.1 there is a "Run Compactor" toggle button in the ACID Transactions setting panel under Hive/Configs.

unknown.png

If an administrator has this enabled, and then uses Service/Actions to add additional Hive Metastores (for Hive Metastore HA), a customer may get the impression "this gets pushed out to all of Metastores and running in multiple places".

I don’t think this can be true because a Hive engineer advised me that we must have exactly 1 compactor initiator in Hive warehouse.

Could someone clarify this please?

[#2 · Accepted solution]

Yes, only one Metastore should run compaction. This is not an issue if there is a standalone Metastore. If there are only embedded metastores within HS2, then only one of them in the cluster should run compaction.

[#3]

Thank you very much for advices.
Then, I will submit idea for Ambari to do either not run multiple Compactor or show some warnings.

[#4]

Embedded metastore currently is not able to run compactions. Standalone metastore is required for this.

[#5]

Each Hive warehouse must have exactly 1 standalone metastore with hive.compactor.initiator.on=true. Ambari has a notion of configuration groups which can be used set this up properly in HA enviroment.