Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

JVM Reuse

avatar
Explorer

Hi

 

I would like to know if there is any drawback to set the following setting bigger than one?

•mapred.job.reuse.jvm.num.tasks
 
Why it set by default to one?
 
Thanks in advance,
1 ACCEPTED SOLUTION

avatar
Super Collaborator

We do not recommend setting it to anything but the default value (1, which means no reuse)

JVM reuse should help with performance becuase it removes the startup lag of the JVM but it is only marginal and comes with a number of drawbacks. Most task will run for a long time (tens of seconds or even minutes) and startup times are not the problem when you look at those run times. When you re-use a JVM there is a change that the heap is not completely clean and it will be fragmented. The fragmentation can lead to more GC's and negate the startup gain. If there is a memory leak it could also affect the memory usage etc.

 

Wilfred

View solution in original post

2 REPLIES 2

avatar
Super Collaborator

We do not recommend setting it to anything but the default value (1, which means no reuse)

JVM reuse should help with performance becuase it removes the startup lag of the JVM but it is only marginal and comes with a number of drawbacks. Most task will run for a long time (tens of seconds or even minutes) and startup times are not the problem when you look at those run times. When you re-use a JVM there is a change that the heap is not completely clean and it will be fragmented. The fragmentation can lead to more GC's and negate the startup gain. If there is a memory leak it could also affect the memory usage etc.

 

Wilfred

avatar
Explorer

Thanks for the answer 🙂