question Re: How to see Mapreduce Spill Disk Activity in Archives of Support Questions (Read Only)

How to see Mapreduce Spill Disk Activity

matt123 — Wed, 01 Mar 2017 03:57:22 GMT

Just started learning Hadoop, I have no idea about as to how to check if a mapreduce job is making spill or not . if so correct me if i am wrong we have to increase io-sort size , please help me out with this.

1 . Also what are all the other parameters that needs to be checked if there is too much spill in mapred-site.xml , hadoop-env.sh files.

Re: How to see Mapreduce Spill Disk Activity

saranvisa — Wed, 01 Mar 2017 05:10:04 GMT

@matt123

Go to link http://ipaddress:8088 and check the Cluster Metrics for the RAM, Container, vcore usage

Also Click on "active nodes" to see the same information by node

Cloudera Manager -> HDFS -> Web UI -> Namenode UI -> See the complete metrics

Re: How to see Mapreduce Spill Disk Activity

matt123 — Wed, 01 Mar 2017 07:05:53 GMT

Thanks for the information.

Does hadoop metrics are collected by default or should we have to enable it. ? Could you please tell me

Also one more quick clarification if there is too much spill in mapreduce job does it mean we have to increase io-sort mb , if so whats an ideal number should be can i start with 1000.

 mapreduce.task.io.sort.mb

Re: How to see Mapreduce Spill Disk Activity

mbigelow — Wed, 01 Mar 2017 20:33:35 GMT

The best indicators are the job counters. Take a look at FILE: Number of bytes written and Spilled Records especially in relation to Map output records. If the spill records are a large portion of the map output records you are spilling a lot.

The first setting below determines how much memory to use for the Map sort and the spill percentage is when it starts spilling to disk as a portion of the first setting. You can tweak both to reduce the amount spilled. The io.sort.mb is a port of the Map heap so there isn't a clear cut "it should be X". You can play around and test it for your job to see how much you can give without slowing down your Mappers from processing data. You could also increase the mapper memory as you increase the io.sort.mb.

mapreduce.task.io.sort.mb
mapreduce.map.sort.spill.percent

Re: How to see Mapreduce Spill Disk Activity

csguna — Thu, 02 Mar 2017 05:54:22 GMT

in mapred-site.xml

mapreduce.map.memory.mb = 
 
mapreduce.task.io.sort.mb =

Re: How to see Mapreduce Spill Disk Activity

matt123 — Thu, 02 Mar 2017 05:58:50 GMT

@mbigelow

@mbigelow - Could you please clarify this - You could also increase the mapper memory as you increase the io.sort.mb.

1 . is it mandatory to increase the mapper memory as we increase io.sort.mb - does it have a dependencies .

2. Say if I increase the mapper memory then follow up I have to increase the
yarn.scheduler.maximum-allocation-mb because of the yarn.nodemanager.vmem-pmem-ratio = 2.1
yarn.nodemanager.resource.memory.mb = 8192

mapreduce.map.java.opts = 2.5GB
mapreduce.map.memory.mb = 3 gb

mapreduce.task.io.sort.mb = 4gb - I can do this .

3. yarn.scheduler.maximum-allocation-mb = 8024 - Will i be able to increase the more than 8GB if I have enough Ram in my system.

Thanks for the help

Re: How to see Mapreduce Spill Disk Activity

matt123 — Thu, 02 Mar 2017 05:57:45 GMT

Thanks

Re: How to see Mapreduce Spill Disk Activity

mbigelow — Thu, 02 Mar 2017 06:09:42 GMT

yarn.scheduler.maximum-allocation-mb - This is the max memory that a single container can get

yarn.nodemanager.resource.memory-mb - This is how much memory per NM is allocated for containers

I always set yarn.scheduler.maximum-allocation-mb eqaul to yarn.nodemanager.resource.memory-mb as the single largest container I could do on a host would be the amount of memory on a host allocated for YARN.

You can set the yarn.scheduler.maximum-allocation-mb to any value, as mention it should not exceed what you set for yarn.nodemanager.resource.memory-mb. If it does, it won't harm anything until someone tries to get a container > yarn.nodemanager.resource.memory-mb.

You might be able to set the configuration like this:

mapreduce.task.io.sort.mb = 4gb - I can do this .

The issue is that the sort buffer is part of the heap of the mapper. For instance a mapper of 3 GB and a heap of 2.5 GB would mean that the sort buffer could quickly fill up the 2.5 GB of heap available. You may not always hit OOM but it is likely due to the poor configuration. In summary yarn.nodemanager.resource.memory-mb > mapreduce.map.memory.mb > mapreduce.task.io.sort. It is not mandatory to increase the mapreduce.map.memory.mb if you increase the sort buffer.

Lets use another example, say you are using 4 GB container with a heap of 3.2 heap. You are spilling a lot of records because you still are using the default sort buffer size. So you increase it to 1 GB. You just shrunk the available memory of your heap from 3.1 GB (3.2 - 100 M, roughly) to 2.2 GB (3.2 - 1). To compensate you could just increase your heap, and along with that your mapper memory. In ths example it would then look like 5 GB container, 4.2 GB heap, and 1 GB sort buffer.

Re: How to see Mapreduce Spill Disk Activity

matt123 — Thu, 02 Mar 2017 09:00:56 GMT

@mbigelow

Thanks for the explanation with example. its clear.

One last clarification

The default - yarn.scheduler.maximum-allocation-mb = 8024 - Will i be able to increase more than 8GB if I have enough Ram in my system.

Re: How to see Mapreduce Spill Disk Activity

mbigelow — Thu, 02 Mar 2017 15:44:19 GMT

Yes. That settings only effect it to put a cap on how large a container can be. It does not mean that your containers will be this size. The yarn.scheduler.minimum-allocation-mb will set the container size if one is not provided by the user.

Re: How to see Mapreduce Spill Disk Activity

matt123 — Thu, 02 Mar 2017 16:07:55 GMT

@mbigelow My English is not that good so I assume from ur answer that I can I set more than 8gb in yarn.scheduler.maximum-allocation-mb please correct me if I am wrong.

Re: How to see Mapreduce Spill Disk Activity

mbigelow — Fri, 03 Mar 2017 02:52:59 GMT

@matt123 You got it.

Re: How to see Mapreduce Spill Disk Activity

matt123 — Fri, 03 Mar 2017 03:22:32 GMT

@mbigelow Cant Thank you engouh Mate