Spark Streaming Architecture

Concepts

• DStream, DStreamGraph（InputDStream, ForEachDStream）

• JobScheduler, JobGenerator

• Receiver, ReceiverSupervisor, BlockGenerator, ReceivedBlockHandler, ReceiverTraker, ReceivedBlockTracker

• Checkpoint, WAL

Links

• Spark Streaming 源码解析系列
• Structured Streaming Programming Guide
• Spark Streaming Programming Guide

---

• Author：HyperJ
• Source：HyperJ’s Blog
• Link：Spark Streaming Architecture