Benefit of DISK_ONLY persists

[#1]

Hi dear experts!

i discovering Spark's persist capabilities and noted interesting behaivour of DISK_ONLY persistance.

as far as i understand the main goal - to store reusable and intermediate RDDs, that were produced from permanent data (that lays on HDFS).

import org.apache.spark.storage.StorageLevel
val input = sc.textFile("/user/hive/warehouse/big_table");
val result = input.coalesce(600).persist(StorageLevel.DISK_ONLY)
scala> result.count()
……
// and repeat command
……..
scala> result.count()

so, i was surprised when saw that second iteration was significantly faster...

could anybody describe why?

thanks!

[#2]

Hm, is that surprising? You described why it is faster in your message. The second time, "result" does not have to be recomputed since it is available on disk. It is the result of a potentially expensive shuffle operation (coalesce)

[#3]

but in the second case I read all dataset as in the first case (without any map operation).

so, in both casese i read whole dataset...

regarding shuffle - i use coalesce instead repartition, so it suppose to avoid shuffle operations...

[#4 · Accepted solution]

The first case is: read - shuffle - persist - count

The second case is: read (from persisted copy) - count

You are right that coalesce does not always shuffle, but it may in this case. It depends on whether you started with more or fewer partitions. You should look at the Spark UI to see whether a shuffle occurred.