@Felix Albani Thank you for answering! Yes, I were monitoring the progress in Spark web UI. The amount of job did not grow from iteration to iteration, while the time spending on these jobs did. To be honest, I managed my problem. I just read the whole file and successfully joined it with another. However, I want to know, why "for"-loops are so ineffective in terms of Spark. ... View more

You can reset the interpreter at the note level. If you go to the note interpreter bindings there is the ability to reset each individual interpreter individually. ... View more

Sorry, it was totally my mistake. I did modify YARN scheduler configuration in a previous tutorial and this affected on spark. I created sub-queues in scheduler, but did not add spark to any of them. ... View more