question Re: Hive database replication with MySQL Cluster in Support Questions

Hive database replication with MySQL Cluster

frank93 — Mon, 07 Nov 2016 22:51:31 GMT

Hi,

I would like to use replication of Hive Metastore database. To do that I installed MySQL Cluster. The problem is that MySQL Cluster needs "ndbcluster" engine for tables to replicate. My default engine is "InnoDB". After converting "hive" tables from "InnoDB" to "ndbcluster" I got strange errors (like "Expected state FINISHED, but found ERROR" or "Error while compiling statement: FAILED: SemanticException java.lang.NullPointerException") when I try to insert data to Hive. Is there any way to workaround this? Doesnt hive support "ndbcluster" engine?

Re: Hive database replication with MySQL Cluster

mjohansson — Tue, 08 Nov 2016 00:24:06 GMT

Hi @Edgar Daeds,

Hive Metsatore is designed for Innodb and do not support MyISAM or ndbcluster as database type in MySQL. You can still replicate your Metastore by using MySQL replication of the binlogs.

Re: Hive database replication with MySQL Cluster

frank93 — Tue, 08 Nov 2016 05:14:51 GMT

@Mats Johansson

Thank you for the answer. I did not mention that, but I want to configure High Availability for MySQL aswell. I am going to try Galera + HAProxy in the immediate future.

Re: Hive database replication with MySQL Cluster

shancz — Thu, 30 Mar 2017 23:36:47 GMT

Any update to your work ?I am working on a similar solution with MySQL cluster and NDBcluster storage engine.

I am planning to place a VIP and a DNS in front of the MySQL cluster to act as a load balancer and single point of entry.

I too had to modify the tables and some column types to be able to import the data from InnoDB.

Regard,

Steven

Re: Hive database replication with MySQL Cluster

frank93 — Fri, 12 May 2017 17:10:03 GMT

I solved the problem by using MySQL master-master replication + MySQL Router as proxy. It works perfect!

Re: Hive database replication with MySQL Cluster

girish28mane — Wed, 17 May 2017 16:40:51 GMT

Hi Edgar,

Do you think MySQL Router is needed when I setup MariaDb galera cluster? As MariaDb galera itself takes care of active server failover.

Re: Hive database replication with MySQL Cluster

charlesbernard — Thu, 09 Nov 2017 10:56:54 GMT

@Mats Johansson, could InnoDB Cluster be used then?