question Re: Runing Compactor against multiple Hive Metastores? in Archives of Support Questions (Read Only)

Runing Compactor against multiple Hive Metastores?

plapierre — Thu, 10 Sep 2015 08:52:57 GMT

Hello Experts,

In Ambari 2.1 there is a "Run Compactor" toggle button in the ACID Transactions setting panel under Hive/Configs.

If an administrator has this enabled, and then uses Service/Actions to add additional Hive Metastores (for Hive Metastore HA), a customer may get the impression "this gets pushed out to all of Metastores and running in multiple places".

I don’t think this can be true because a Hive engineer advised me that we must have exactly 1 compactor initiator in Hive warehouse.

Could someone clarify this please?

Re: Runing Compactor against multiple Hive Metastores?

gvetticaden1 — Thu, 10 Sep 2015 08:54:36 GMT

Yes, only one Metastore should run compaction. This is not an issue if there is a standalone Metastore. If there are only embedded metastores within HS2, then only one of them in the cluster should run compaction.

Re: Runing Compactor against multiple Hive Metastores?

gvetticaden1 — Thu, 10 Sep 2015 09:00:59 GMT

Thank you very much for advices.
Then, I will submit idea for Ambari to do either not run multiple Compactor or show some warnings.

Re: Runing Compactor against multiple Hive Metastores?

ekoifman — Thu, 10 Dec 2015 01:12:49 GMT

Embedded metastore currently is not able to run compactions. Standalone metastore is required for this.

Re: Runing Compactor against multiple Hive Metastores?

ekoifman — Thu, 10 Dec 2015 01:16:30 GMT

Each Hive warehouse must have exactly 1 standalone metastore with hive.compactor.initiator.on=true. Ambari has a notion of configuration groups which can be used set this up properly in HA enviroment.