question Re: Deploying hadoop cluster in Archives of Support Questions (Read Only)

Deploying hadoop cluster

kabirdas_bk — Mon, 01 Feb 2016 17:32:37 GMT

I want to set up hadoop cluster where i want master node and few slave node to be in some country and remaining slave nodes sitting in different country. Is it possible to deploy this? Is there any article or tutorial which might help me to get the solution.

Re: Deploying hadoop cluster

Shelton — Mon, 01 Feb 2016 17:42:46 GMT

Of course that possible. This doc could help you Topology scripts are used by Hadoop to determine the rack location of nodes

Topology configuration

Re: Deploying hadoop cluster

pminovic — Mon, 01 Feb 2016 18:04:00 GMT

Hi @Kabirdas B K yes that's possible, you can just add remote nodes like local nodes, and yes you can set rack awareness, but it's not recommended because the performance, for example when running jobs can be very poor. Here is one such experience.

Re: Deploying hadoop cluster

bleonhardi — Mon, 01 Feb 2016 18:44:16 GMT

This! You can but you shouldn't. Cross-datacenter hadoop clusters do not perform. Don't do it.

Use data replication software like DistCP or Falcon to sync clusters in different data centers instead.

Re: Deploying hadoop cluster

aervits — Mon, 01 Feb 2016 19:10:18 GMT

Do not do it! @Kabirdas B K

Re: Deploying hadoop cluster

nsabharwal — Mon, 01 Feb 2016 19:15:07 GMT

@Kabirdas B K

I don't recommend doing this. It's possible but not the best practice. You can setup DR using Falcon or WanDisco.