question Re: How can we keep the data in 2 data centers in sync ? in Support Questions

How can we keep the data in 2 data centers in sync ?

ARVINDR — Wed, 04 Mar 2020 14:13:20 GMT

Hi Team,

We have 12 nodes cluster hosted on premise in 2 different regions.

The question is how can we keep the data in 2 data centers in sync and what will be the latency.

Appreciate your help.

Thanks & Regards

Re: How can we keep the data in 2 data centers in sync ?

StevenOD — Wed, 04 Mar 2020 08:57:28 GMT

Hi @ARVINDR

I'd like to clarify your scenario. Do you have

A) 12 nodes in region A and 12 nodes in region B i.e. 2 distinct Cloudera clusters with a total of 24 nodes and you want to replicate data between these clusters?

B) 12 nodes split across regions A and B i.e. a single cluster of 12 nodes?

Also what version of Cloudera are you using?

Regards,

Steve

Re: How can we keep the data in 2 data centers in sync ?

ARVINDR — Wed, 04 Mar 2020 13:00:28 GMT

Hi Steven,

Pls consider scenario A

A) 12 nodes in region A and 12 nodes in region B i.e. 2 distinct Cloudera clusters with a total of 24 nodes and we want to replicate data between these clusters

We are using version HDP 3.0.1

Thanks & Regards

Re: How can we keep the data in 2 data centers in sync ?

StevenOD — Wed, 04 Mar 2020 14:43:36 GMT

Hi @ARVINDR

In this case, you would use Data Lifecycle Manager to replicate data between the two clusters.

Here is a link to the documentation:

https://docs.cloudera.com/HDPDocuments/DLM1/DLM-1.5.1/administration/content/dlm_hdfs_replication_overview_new.html

The latency will be a function of your network. I can share some general networking guidelines here:

https://docs.cloudera.com/documentation/other/reference-architecture/topics/ra_bare_metal_deployment.html#concept_qlk_mpk_v2b

Regards,

Steve

Re: How can we keep the data in 2 data centers in sync ?

ARVINDR — Mon, 09 Mar 2020 12:28:34 GMT

Hi @StevenOD,

Thanks for the details.

Just one query, we are building Hadoop on top of Isilon , will the following still holds true in that case ?

Thanks & Regards

Arvind.

Re: How can we keep the data in 2 data centers in sync ?

StevenOD — Wed, 11 Mar 2020 13:37:14 GMT

Hi @ARVINDR ,

I am an expert in Isilon but I'm not sure that Data Lifecyle Manager (DLM) supports using Isilon as the storage layer.

Isilon uses the OneFS file system. OneFS supports its own utilities for backing up data and replication so it might be better to use tools that are native to Isilon in this scenario.

Regards,

Steve

Re: How can we keep the data in 2 data centers in sync ?

ARVINDR — Thu, 12 Mar 2020 02:07:49 GMT

Thanks @steve

Could you pls suggest native tools in this scenario ?

Thanks & Regards

Re: How can we keep the data in 2 data centers in sync ?

StevenOD — Thu, 12 Mar 2020 09:28:54 GMT

Hi @ARVINDR ,

I suggest you reach to Dell EMC for guidance around Isilon.

I found some documentation here:

https://www.dellemc.com/ro-ro/collaterals/unauth/white-papers/products/storage/h10588-isilon-data-availability-protection-wp.pdf

that suggests some techniques. I'll caveat that I'm not an expert in Isilon so my responses here are best endeavors.

Regards,

Steve

Re: How can we keep the data in 2 data centers in sync ?

ARVINDR — Thu, 12 Mar 2020 10:47:44 GMT

Thanks for sharing documents @StevenOD