Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Joining large tables (dataframe and sql), but only want few columns: select before or after join?

Explorer

Concerning memory usage and efficiency, when joining two large tables with many columns but only want a few columns from each of them, is it better to select() before or after the join()? My instinct tells me to select() before the join but some other voices would be very helpful

1 ACCEPTED SOLUTION

Explorer

Yes. Reducing size of dataset before JOIN would surely help rather than other way round.

View solution in original post

4 REPLIES 4

Explorer

Yes. Reducing size of dataset before JOIN would surely help rather than other way round.

Explorer

Does this also hold for other methods besides JOIN? E.g., I want to do a groupBy. Should I select() before the groupBy()?

Yes. You can think of select() as the "filter" of columns where filter() filters rows. You want to reduce the impact of the shuffle as much as possible. Perform both of these as soon as possible. The groupBy() is going to cause a shuffle by key (most likely). Be careful with the groupBy(). If you can accomplish what you need to do with a reduceBy(), you should use that instead.

If you mean dataframe instead of dataset, SparkSQL will handle much of this optimization for you. But if using normal RDDs, you are going to have to deal with these types of optimizations on your own.

Explorer

Yes. A projection before any sort of transformation/action would help in computation time and storage optimization.