Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark sql join decide shuffle partitions

spark sql join decide shuffle partitions

New Contributor

Hi Team,

 

In spark sql, when we join two tables whether it is big or small tables, the default number of shuffle partitons results 200. How we can derive the number of shuffle partitons and set  number of shuffles before joining two tables in spark sql

 

And another doubt when we join table A and table B , both table utilize 200 partitions or each table utilizing 200 + 200 = 400 partitions. how it works? 

 

Could you help us solving this?

Don't have an account?
Coming from Hortonworks? Activate your account here