question Re: MapReduce 2 Optimization in Heterogeneous Cluster in Archives of Support Questions (Read Only)

MapReduce 2 Optimization in Heterogeneous Cluster

jalil1408 — Fri, 16 Sep 2022 09:57:49 GMT

I have this configuration:

    Hadoop: v2 (Yarn)
    An input file: Size = 100 GB.
    3 Slaves: each has 4 VCORES with Speed = 2 GHz and RAM = 8 GB
    5 Slaves: each has 2 VCORES with Speed = 1 GHz and RAM = 2 GB
    MapReduce program: WordCount

How can I minimize WordCount execution time by assigning small input splits to the 5 slower slaves and big input splits to the 3 fastest slaves?

Re: MapReduce 2 Optimization in Heterogeneous Cluster

Wilfred — Thu, 21 Jan 2016 03:12:11 GMT

You need to setup the nodes with the proper vcores and memory available for the NM. That should solve the problem. It will put more load on the larger nodes than on the small nodes.

The container is also scheduled on the node based on the data locality which is out of your control.

You can however not say start processing of the split on a specific node.

Wilfred

Re: MapReduce 2 Optimization in Heterogeneous Cluster

jalil1408 — Thu, 21 Jan 2016 08:54:13 GMT

I wonder why YARN doesn't support VCORE SPEED in container configuration!

Re: MapReduce 2 Optimization in Heterogeneous Cluster

Wilfred — Sun, 24 Jan 2016 23:09:53 GMT

A vcore is a virtual core. You can define it however you want.

You could, as an example, define that a vcore is the processing power that is delivered by a 1GHz thread core. A 3GHz core would than be comparable to 3 vcores in the node manager. Your container request then needs to use multiple vcores which handles the difference in speed.

Not a lot of clusters do this due to the administrative overhead and the fact that if the end users do not use the vcore correctly it can overload the faster machines.

Wilfred