Mix on premise and cloud nodes

[#1]

Hi,

I've read the following forumn post (http://community.cloudera.com/t5/Cloudera-Director-Cloud-based/Mix-cloud-and-on-premise-Data-nodes/t...) that states if mixing a cluster with both on prem and cloud bases nodes network latency could be a problem.

I'm trying to find a little more information around what Cloudera would recommend with regards to latency.  

I currently have a small cluster on prem, however we also have a direct connect to AWS which i'd like to explore the option of adding new EC2 nodes to the cluster.

Network latancy is the following:

Office to on prem nodes: 6ms

Office to EC2 nodes: 16ms

On-Prem nodes to EC2: 10ms

EC2 to on prem nodes: 10ms

Given the latency seems relativly small, is there any disadvantage to using cloud with on prem?

[#2 · Accepted solution]

@chriswalton007

There are different types of latencies 

Namenode RPC latency
Journal Node FSync latency
network latency
etc

There are few points here,

1. Your network latency will vary based on the traffic in your cluster. It may create trouble during peak hours.

2. The latency issue may leads to follows: As we know, the master daemons will always wait for the update from child daemons in every few seconds, and master will consider the child is not available/dead in case of any delay and look for alternate. It is unnecessary unless it is a real issue with child.

3. As far as NN RPC is the concern, in a HA cluster, both active and standby NN has to talk each other and it should be in sync with in few seconds. If they are not in sync and if something went wrong on active NN, the standby become active but it may not be up to date and it will lead to confusion.

end of the day, every seconds are matter in a distributted cluster. 

But in your use case, not sure you are going to use cloudera director if so, the link that you have shared says it will not allow to create a mixed cloud/on-premise cluster. But if you are going to use a different tool and it will allow you to configure the mixed cloud/on-prem then you can go ahead based on the below...

1. If you are going to try this for a non-prod environment first

2. if you have less work loads