Clusters in multiple regions managed single Manager/Director

[#1]

Wondering what is the design guidance for setting up clusters in different AWS Regions. Is it recommended to have a local Manager/Director in each region or have a central one and have the other regions talk to that central Manager/Director?

[#2 · Accepted solution]

A single Cloudera Director instance can have multiple environments, potentially spanning multiple cloud providers or regions. Each environment is configured with a specific cloud provider and region. A single environment can have multiple deployments, each corresponding to a single Cloudera Manager instance. Each Cloudera Manager instance can manage multiple clusters.

The only requirement is that Cloudera Director can reach the hosts it creates via SSH. If you use Cloudera Director to manage clusters within a single region, you can potentially have all communication be on the private IP addresses of the instances. If you use Cloudera Director to manage clusters in a different cloud provider or region, you will either have to do special networking (e.g. VPN) to gain access to the private IP addresses, or you will have to permit SSH access on the public IP address, which has security implications.

[#3]

We are having a cluster in Azure US region and we are going to build a new hadoop cluster in EU region. We are planning to install this new cluster(EU region) using same same cloudera manager server which is there in US region. Will it work?

[#4]

The feature that John described above does not provide this functionality, and Director doesn't provide any way to make this work. It would be simple to create a new CM server (deployment) on the EU region and add the cluster to that, of course, but not to add that cluster to the US deployment.

It is technically *possible* to create a cluster this way, but this would need to be accomplished outside of Director. It would require some network configuration that may be inadvisable, though I'm not an expert here and it's not something we'd be able to support of help with.

[#5]

Sorry to drag up an old topic but I am hoping there is now some official guidance from Cloudera - or perhaps a reference architecture. 

As soon as an organisation builds a Cloudera EDH cluster in one Amazon AWS region (or Azure or Google Cloud, or whatever) then they soon realise they might want another cluster in a different region or a different Availability Zone. 

Personally I am happy sticking with one Region but multiple AZs - so I am thinking that I want one Cloudera Director, one Cloudera Manager, and separate clusters in each AZ.