@Felix thanks for your input. Shuffle data is serialized over the network so when deserialized its spilled to memory ==> From my understanding, operators spill data to disk if it does not fit in memory. So it's not directly related to the shuffle process. Furthermore, I have plenty of jobs with shuffles where no data spills. Shuffle spill (memory) is the size of the deserialized form of the data in memory at the time when we spill it, whereas shuffle spill (disk) is the size of the serialized form of the data on disk after we spill it. This is why the latter tends to be much smaller than the former ==> In the present case the size of the shuffle spill (disk) is null. So I am still unsure of what happened to the "shuffle spilled (memory) data" ... View more