I faced a similar issue with the following error 18/12/10 11:33:12 ERROR YarnScheduler: Lost executor 2 on server1: Container marked as failed: container_e06_1544075636158_0018_01_000003 on host: server1. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. Killed by external signal The code was working fine with the default partition value of 19 but when partitionBy method was introduced and newHashpartitioner was increased to 38 , it throws the above error but the job was still running with no progress ... View more