Shockwave
Shockwave is a memory-safe hierarchical planner for hybrid-parallel Transformer training.
Shockwave is a memory-safe hierarchical planner for hybrid-parallel Transformer training. It combines memory-feasible parallel configuration search, sensitivity-guided configuration selection, and pipeline-stage layer partitioning to generate an efficient training plan for a target model and GPU cluster.
The implementation is available in the Shockwave GitHub repository. The accompanying paper, Memory-Safe Hierarchical Planning for Hybrid-Parallel Transformer Training, was published in ACM Transactions on Architecture and Code Optimization.