What if MySQL Metadata Host Failed?

[#1]

Hi, I have a general question. All of our metadata (CM, Hive MS, Oozie, Navigator, etc) is stored in a local MySQL instance on the Cloudera Manager host (see list of DB's below). If this host were to fail, or the MySQL service were to become unavailable for a period of time, would all of the core cluster services (excluding CM services) continue to function?  I am unclear if we need to look at implementing a MySQL cluster to maintain cluster uptime.  Thank you!

Edit: I should add that we are currently replicating MySQL to a slave.

amon
rman
metastore
metastore
sentry
nav
navms
oozie
sqoop
hue

[#2 · Accepted solution]

@DataMike,

If MySQL is out of service, all roles that need to use MySQL will stop to function, this includes roles like CM itself, Hive, Sentry, Oozie, Hue etc. HDFS and YARN should still function.

You should have Database HA setup, as mentioned here:
https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_dbms.html

So if MySQL master is down, at least the read from Slave can still function, while you fix the master. Or have the MySQL backup and replicated to another MySQL instance that is ready to take over should the current one fail.

Cheers
Eric

[#3]

Thank you @EricL .  Makes sense.  We currently have MySQL set up in a master/slave config.  But it looks like even with that... if the Master is offline, you have to manually edit the CM config to point to the slave.  There is no where in CM (that I see) to add a slave/secondary MySQL hostname.

So I assume that is why you reference setting up MySQL in a "cluster" configuration and not just master/slave replication - correct?  Thank you.

[#4]

Hi @DataMike,

Yes, once CM is down, you can't update the configuration unless you "hack" into CM's backend database, which is also down in our scenario.

So if you setup MySQL in such a way that CM will connect to LB / VIP host, which will route to the MySQL host that you want to, so in the event of one MySQL instance is down, simply update LB to only go to the working one. This avoids the needs to update the CM config.

Hope that can help.

Cheers
Eric