Skip to main content
Mannoj
Silver ⭐️⭐️
June 26, 2023
Question

Hive Mysql Metastore Row lock increases

Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster).

 

select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update

 

> Hive is a framework, we couldn't get the flow to understand also can't alter the logic from app end.

 

> Looking for solutions like any version that works in Mysql/Postgresql or any Metastore DBs that can scale for Hive Metastore without lockign issues.

 

> Also looking for any work around solution from DB end for this Mysql Row Locking.

6 replies

cjervis
Community Manager
Community Manager
June 26, 2023

Welcome to the community @Mannoj. I'm not an expert but while you wait for one, I found this article/video on Hive Locking Mechanism. Hopefully it moves you closer in the right direction. 

Translate
Keep the questions coming
Mannoj
MannojAuthor
Silver ⭐️⭐️
June 28, 2023

@cjervis These are locks in Hive, we have a mysql backend for metastore and that keeps locking in above mentioned query. Whenever the row locks increase at mysql end we will face issues in hive in processing queries and backlog builds.

Translate
cjervis
Community Manager
Community Manager
June 28, 2023

Thank you for the clarification. Hopefully someone will be able to assist.

Translate
Keep the questions coming
smruti
Gold ⭐️⭐️⭐️
Gold ⭐️⭐️⭐️
July 4, 2023

@Mannoj RDBMS HA for metastore is not officially supported yet. We have a knowledge article on the same. Cloudera's statement on RDBMS HA can be found here.

Translate
Mannoj
MannojAuthor
Silver ⭐️⭐️
August 12, 2024

Just got the article access. By the way this issue occurs in mysql even not a galera cluster types. even i make the same Mariadb as master / slave, the same issue occurs, so this is not the problem with galera cluster, is for sure.

Translate
Mannoj
MannojAuthor
Silver ⭐️⭐️
July 19, 2023
I dont have cloudera enterprise login. So can’t check. Such KA can be open
to public.
Translate