The following has a good overview: Essentially RDDs are directly implemented code. Whatever you write gets executed. Dataframes on the other hand get compiled into an execution plan and then executed by the same engine. Essentially there is only a Dataframe API in python. So you would expect the same performance unless you use big heavy python udfs. https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html Essentially Dataframes have two advantages over RDDs. A) Most people do not know how to write optimized Spark code and B) the optimizer can do some tricks based on data charateristics that a user might not be aware of during write time. ( dataset size etc. ) https://0x0fff.com/spark-dataframes-are-faster-arent-they/ Edit: I was curious and did dig a bit deeper and I think here is the best overview, essentially as said, as long as you use the basic Dataframe functions performance is equal because the Dataframe code ( Python or Scala ) gets translated into the same code RDD code ( scala ) but if you use heavy python udfs you will see performance differences again. https://ogirardot.wordpress.com/2015/05/29/rdds-are-the-new-bytecode-of-apache-spark/ "This is still true if you want to use Dataframe’s User Defined Functions, you can write them in Java/Scala or Python and this will impact your computation performance – but if you manage to stay in a pure Dataframe computation – then nothing will get between you and the best computation performance you can possibly get."
... View more
