JVM Reuse

[#1]

Hi

I would like to know if there is any drawback to set the following setting bigger than one?

•mapred.job.reuse.jvm.num.tasks

Why it set by default to one?

Thanks in advance,

[#2 · Accepted solution]

We do not recommend setting it to anything but the default value (1, which means no reuse)

JVM reuse should help with performance becuase it removes the startup lag of the JVM but it is only marginal and comes with a number of drawbacks. Most task will run for a long time (tens of seconds or even minutes) and startup times are not the problem when you look at those run times. When you re-use a JVM there is a change that the heap is not completely clean and it will be fragmented. The fragmentation can lead to more GC's and negate the startup gain. If there is a memory leak it could also affect the memory usage etc.

Wilfred

[#3]

Thanks for the answer 🙂