Support Questions
Find answers, ask questions, and share your expertise

What is the need to turn off speculative execution?

Why it is the need to turn off speculative execution?


The main work of speculative execution is to reduce the job execution time; however, the clustering efficiency is affected due to duplicate tasks. Since in speculative execution redundant tasks are being executed, thus this can reduce overall throughput. For this reason, some cluster administrators prefer to turn off the speculative execution in Hadoop.

@Dukool SHarma

I can think of these 2 cases as a reason why you wont want to have speculative execution on: resource limitation and duplicate output results when saving to database/sink.

Speculative execution will not stop the slow executor currently running the task and actually launch the task in new executor to do the same processing hopefully faster. Which ever finishes first wins. This leads to more resource utilization.

Also if you are saving to a database, for example, this could also lead into duplicate information at the DB side since there are 2 executors doing same processing eventually.


*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.