@mbigelow Cant Thank you engouh Mate ... View more

@mbigelow My English is not that good so I assume from ur answer that I can I set more than 8gb in yarn.scheduler.maximum-allocation-mb  please correct me if I am wrong.  ... View more

@mbigelow Thanks for the explanation with example. its clear. One last clarification  The default - yarn.scheduler.maximum-allocation-mb   = 8024 - Will i  be able to increase more than 8GB if I have enough Ram in my system. ... View more

Thanks ... View more

@mbigelow    @mbigelow   - Could you please clarify this -  You could also increase the mapper memory as you increase the io.sort.mb.    1 . is it mandatory to increase the mapper memory as we increase io.sort.mb - does it have a  dependencies .    2. Say if I increase the mapper memory then follow up I have to increase the  yarn.scheduler.maximum-allocation-mb  because of the  yarn.nodemanager.vmem-pmem-ratio = 2.1 yarn.nodemanager.resource.memory.mb = 8192   mapreduce.map.java.opts = 2.5GB mapreduce.map.memory.mb = 3 gb   mapreduce.task.io.sort.mb = 4gb - I can do this .     3. yarn.scheduler.maximum-allocation-mb   = 8024 - Will i  be able to increase the more than 8GB if I have enough Ram in my system.   Thanks for the help ... View more

Thanks for the information.  Does hadoop metrics are collected by default or should we have to enable it. ? Could you please tell me    Also one more quick clarification if there is too much spill in mapreduce job does it mean we have to increase io-sort mb , if so whats an ideal number should be can i start with 1000.    mapreduce.task.io.sort.mb ... View more