Quick question. Have you tried without the broadcast? 1.5 million records is not that small, that you should send it to 800 executors. Also shouldn't you be doing something with the joinDF, e.g. at least a joinDF.count()? The join by "key" looks interesting as well. Have you considered trying you logic with a smaller dataset first, say 8000 records? ... View more