question How to utilize infiniband backbone during MapReduce. in Archives of Support Questions (Read Only)

How to utilize infiniband backbone during MapReduce.

brian_ramsel — Tue, 22 Dec 2015 03:22:10 GMT

We are currently trying to use the phoenix csv bulk loader mapreduce tool. It is taking about a hour and a half for a 170 GB csv. The map is usally done quickly but the reduce seems to be taking much longer than it should. I am believe the fact we are utilizing a 1 Gb is a contributing factor to this. We have some old 10 Gb infiniband equipment laying around and I was considering trying to implement this as the backbone of HDFS and MapReduce. I have come across two articles mentioning multihoming, neither of which I believe gives me enough detail to solve this problem. Any documentation or direction is greatly appreciated.

Re: How to utilize infiniband backbone during MapReduce.

nsabharwal — Tue, 22 Dec 2015 03:22:47 GMT

@jeff @Paul Codding

Re: How to utilize infiniband backbone during MapReduce.

aervits — Thu, 24 Dec 2015 22:15:14 GMT

this would make an excellent wiki post

Re: How to utilize infiniband backbone during MapReduce.

bsaini — Fri, 25 Dec 2015 01:08:31 GMT

@Brian Ramsel

Have to explored and tried changing the memory setting for reducer and the number of reducers?

I am totally in agreement with moving to 10G but just wondering if there is an opportunity to improve the performance with current setup. In the recent past, working with a prospect on a POC, we were able to ingest 600GB file in about 30 mins on a small 4 node cluster. (64GB RAM, 10GiBE, other tuning done at the app/service level). Not sure how big this cluster is and what is the hardware spec though.

Re: How to utilize infiniband backbone during MapReduce.

brian_ramsel — Tue, 29 Dec 2015 03:29:16 GMT

Sorry I haven't responded yet been out of the office with the holidays. From what I can tell the reduce memory is set to 5GB. I am unsure about the number of reduces. We have an 8 node cluster each node has 16 cores and 192 GB of RAM.