Hive database replication with MySQL Cluster

[#1]

Hi,

I would like to use replication of Hive Metastore database. To do that I installed MySQL Cluster. The problem is that MySQL Cluster needs "ndbcluster" engine for tables to replicate. My default engine is "InnoDB". After converting "hive" tables from "InnoDB" to "ndbcluster" I got strange errors (like "Expected state FINISHED, but found ERROR" or "Error while compiling statement: FAILED: SemanticException java.lang.NullPointerException") when I try to insert data to Hive. Is there any way to workaround this? Doesnt hive support "ndbcluster" engine?

[#2 · Accepted solution]

Hi @Edgar Daeds,

Hive Metsatore is designed for Innodb and do not support MyISAM or ndbcluster as database type in MySQL. You can still replicate your Metastore by using MySQL replication of the binlogs.

[#3]

@Mats Johansson

Thank you for the answer. I did not mention that, but I want to configure High Availability for MySQL aswell. I am going to try Galera + HAProxy in the immediate future.

[#4]

@Mats Johansson, could InnoDB Cluster be used then?

[#5]

Hi

Any update to your work ?I am working on a similar solution with MySQL cluster and NDBcluster storage engine.

I am planning to place a VIP and a DNS in front of the MySQL cluster to act as a load balancer and single point of entry.

I too had to modify the tables and some column types to be able to import the data from InnoDB.

Regard,

Steven

[#6]

I solved the problem by using MySQL master-master replication + MySQL Router as proxy. It works perfect!

[#7]

Hi Edgar,

Do you think MySQL Router is needed when I setup MariaDb galera cluster? As MariaDb galera itself takes care of active server failover.