Beyond the two pieces of code( map and reduce phase), we have combiner,shuffle and sort phase. If failure occurs after each phase, output from each phase will be affected , so each phase will execute again from scratch. So I have created a checkpoint after each phase.(map,combine,shuffle and sort,reduce) by finding out the each phase cost time. I have created a condition like after each phase cost time, launch checkpoint. Is this condition correct? I want to know whether this idea is possible???
... View more