Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is the need to turn off speculative execution?

What is the need to turn off speculative execution?

New Contributor

Why it is the need to turn off speculative execution?

2 REPLIES 2
Highlighted

Re: What is the need to turn off speculative execution?

New Contributor

The main work of speculative execution is to reduce the job execution time; however, the clustering efficiency is affected due to duplicate tasks. Since in speculative execution redundant tasks are being executed, thus this can reduce overall throughput. For this reason, some cluster administrators prefer to turn off the speculative execution in Hadoop.

Re: What is the need to turn off speculative execution?

@Dukool SHarma

I can think of these 2 cases as a reason why you wont want to have speculative execution on: resource limitation and duplicate output results when saving to database/sink.

Speculative execution will not stop the slow executor currently running the task and actually launch the task in new executor to do the same processing hopefully faster. Which ever finishes first wins. This leads to more resource utilization.

Also if you are saving to a database, for example, this could also lead into duplicate information at the DB side since there are 2 executors doing same processing eventually.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.