Spark Shuffle

Concept

• Task(ShuffleMapTask, ResultTask)

Shuffle Write

• ShuffleMapTask#runTask -> ShuffleManager(SortShuffleManager)#getWriter -> ShuffleWriter(SortShuffleWriter, UnsafeShuffleWriter)#write

• ShuffleWriter

  SortShuffleWriter(ExternalSorter, Aggregator, ExternalAppendOnlyMap)

  UnsafeShuffleWriter(ShuffleExternalSorter)

Shuffle Read

• ShuffledRDD#compute -> ShuffleManager(SortShuffleManager)#getReader -> ShuffleReader(BlockStoreShuffleReader)#read

• BlockStoreShuffleReader(ExternalSorter, Aggregator, ExternalAppendOnlyMap)

Links

• Spark Architecture: Shuffle
• Shuffle 过程
• 详细探究Spark的shuffle实现
• Spark Shuffle的技术演进
• Spark Documentation

---

• Author：HyperJ
• Source：HyperJ’s Blog
• Link：Spark Shuffle