Chorus

Chorus is a compiler-runtime framework that globally schedules transient parameter materialization for memory-constrained sharded LLM training.

Chorus is a compiler-runtime framework for transient parameter materialization in memory-constrained sharded large language model training. It treats a complete training iteration as a coupled scheduling problem, coordinating communication, full-parameter buffer lifetimes, and computation under time-varying GPU memory headroom.

Chorus workflow from PyTorch graph capture through global scheduling and runtime persistent retention

As the overview shows, the compiler builds a whole-iteration graph, derives a profile-guided block-level model, selects all-gather launch positions and retention decisions, and realizes the result through graph rewriting and fused prefetching. Runtime warmup then finalizes a persistent retention set using measured memory behavior.

The public implementation supports two distributed training backends: DeepSpeed ZeRO-3 through DeepCompile and PyTorch SimpleFSDP through compiled autograd. The source code and usage guide are available in the Chorus GitHub repository.