this looks like you have data skew issue, meaning your group by key is skewed, resulting in unbalanced data between partitions. you can inspect your key distribution, if skewness is real, you need to change key or add salt into the groupby so data can be evenly distributed. ... View more

thanks David, planning to create one template and done, as I see no advantage really of 50 templates ... View more

Thanks Neeraj, saw the link before, increased memory like they said, but still see errors like above.And also they dont answer about data loss, reprocessing the data etc . ... View more