question Re: Hive Mysql Metastore Row lock increases in Support Questions

Hive Mysql Metastore Row lock increases

Mannoj — Mon, 26 Jun 2023 15:38:27 GMT

Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster).

select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update

> Hive is a framework, we couldn't get the flow to understand also can't alter the logic from app end.

> Looking for solutions like any version that works in Mysql/Postgresql or any Metastore DBs that can scale for Hive Metastore without lockign issues.

> Also looking for any work around solution from DB end for this Mysql Row Locking.

Re: Hive Mysql Metastore Row lock increases

cjervis — Mon, 26 Jun 2023 16:34:50 GMT

Welcome to the community @Mannoj. I'm not an expert but while you wait for one, I found this article/video on Hive Locking Mechanism. Hopefully it moves you closer in the right direction.

Re: Hive Mysql Metastore Row lock increases

Mannoj — Wed, 28 Jun 2023 13:56:23 GMT

@cjervis These are locks in Hive, we have a mysql backend for metastore and that keeps locking in above mentioned query. Whenever the row locks increase at mysql end we will face issues in hive in processing queries and backlog builds.

Re: Hive Mysql Metastore Row lock increases

cjervis — Wed, 28 Jun 2023 14:25:22 GMT

Thank you for the clarification. Hopefully someone will be able to assist.

Re: Hive Mysql Metastore Row lock increases

smruti — Tue, 04 Jul 2023 18:57:26 GMT

@Mannoj RDBMS HA for metastore is not officially supported yet. We have a knowledge article on the same. Cloudera's statement on RDBMS HA can be found here.

Re: Hive Mysql Metastore Row lock increases

Mannoj — Wed, 19 Jul 2023 07:38:41 GMT

I dont have cloudera enterprise login. So can’t check. Such KA can be open
to public.

Re: Hive Mysql Metastore Row lock increases

Mannoj — Mon, 12 Aug 2024 11:46:44 GMT

Just got the article access. By the way this issue occurs in mysql even not a galera cluster types. even i make the same Mariadb as master / slave, the same issue occurs, so this is not the problem with galera cluster, is for sure.