Pipeline parallelism
Two-stage pipeline parallelism The forward pass sends activations from stage zero to stage one. The backward pass sends activation gradients from stage one back to stage zero. STAGE BOUNDARY GPU 0 · stage 0 layer 1 parameters W₁ layer 2 parameters W₂ GPU 1 · stage 1 layer 3 parameters W₃ layer 4 parameters W₄ microbatch x₀ loss L₀ SEND ACTIVATION a₂ → loss ∂L input ∂x₀ SEND INPUT GRADIENT ← ∂L/∂a₂