Publications | Zuoyan Zhang

2026

TACO 2026
Memory-Safe Hierarchical Planning for Hybrid-Parallel Transformer Training

Zuoyan Zhang, Zhiqiang Zhang, Nanqin Wang, Jinyi Wang, and 1 more author

ACM Trans. Archit. Code Optim., Aug 2026

Abs DOI Bib PDF

Training decoder-style Transformer models at scale requires composing data, tensor, pipeline, context, and, for sparse models, expert parallelism under tight device memory limits. Without a reliable pre-profiling admission model, algebraically legal but memory-infeasible configurations can consume profiling budget without yielding useful throughput measurements. Choosing a pipeline degree also leaves a structured layer-to-stage mapping problem unresolved. This paper presents Shockwave, a hierarchical planner that operates through the exposed launcher and partition interfaces of Megatron-style dense and sparse Transformer runtimes. Shockwave first builds a conservative memory-feasible region with an analytical model and backend-calibrated safety margins. It then profiles a balanced baseline and a few single-dimension anchors to learn which parallel degrees improve throughput most on the target runtime and uses that ranking to select a global strategy. Finally, it solves a mixed-integer program for stage-aware layer allocation under a fixed 1F1B pipeline schedule. Implemented as a thin control layer on top of Megatron-LM and MindSpeed, Shockwave matches or exceeds expert-curated plans, achieving geometric mean throughput gains of 1.08x over expert-tuned Megatron hybrid configurations on GPUs and 1.04x over expert MindSpeed recipes on NPUs. Relative to the evaluated automatic baselines, it improves throughput by 1.33x on GPUs and 1.10x on NPUs while reducing configuration search time by 4.59x and 5.60x.
@article{shockwave2026, author = {Zhang, Zuoyan and Zhang, Zhiqiang and Wang, Nanqin and Wang, Jinyi and Zhao, Jie}, title = {Memory-Safe Hierarchical Planning for Hybrid-Parallel Transformer Training}, year = {2026}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, issn = {1544-3566}, url = {https://doi.org/10.1145/3842760}, doi = {10.1145/3842760}, journal = {ACM Trans. Archit. Code Optim.}, month = aug, keywords = {large language model training, automatic parallelism, hybrid parallelism, memory-feasibility modeling, pipeline parallelism, execution-plan synthesis}, }
TACO 2026
Global Scheduling of Transient Parameter Materialization for Sharded Training

Zuoyan Zhang, Jinyi Wang, Zhuo Tang, and Jie Zhao

ACM Trans. Archit. Code Optim., Aug 2026

Abs DOI Bib PDF

Parameter-sharded training reduces device-memory use by storing model parameters as shards and reconstructing full parameters only before the computations that consume them. This materialization is implemented through frequent all-gathers, which can become exposed waits in each forward-backward training iteration. Hiding these waits is not only a communication-volume problem. It requires deciding where to launch each all-gather and how long the materialized full-parameter buffer should remain live. We present Chorus, a compiler-runtime framework for transient parameter materialization in memory-constrained sharded training. Chorus lowers the forward and backward passes into a whole-iteration graph that makes all-gathers, use-site waits, and releases explicit. It then builds a profile-guided block model and solves a mixed-integer scheduling problem that jointly chooses all-gather launch positions and iteration-local retention decisions under time-varying memory headroom. We implement Chorus on top of torch.compile and evaluate it on 7B to 13B dense Transformer models and one MoE model across 4 to 16 A100 GPUs. Chorus improves throughput over state-of-the-art ZeRO-style and FSDP-style baselines while staying within the configured memory budget.
@article{chorus2026, author = {Zhang, Zuoyan and Wang, Jinyi and Tang, Zhuo and Zhao, Jie}, title = {Global Scheduling of Transient Parameter Materialization for Sharded Training}, year = {2026}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, issn = {1544-3566}, url = {https://doi.org/10.1145/3838601}, doi = {10.1145/3838601}, journal = {ACM Trans. Archit. Code Optim.}, month = aug, article-number = {3838601} }

HLPP 2026

Compiler-Guided Polynomial-Level Parallelism for FHE-Encrypted Machine Learning Inference

Zijian Duan, Zuoyan Zhang, Tao Dai, Zhihong Liang, and 1 more author

In Proceedings of the 19th International Symposium on High-Level Parallel Programming and Applications, Paris, France, Jul 2026

Bib PDF

@inproceedings{hlpp2026,
  author = {Duan, Zijian and Zhang, Zuoyan and Dai, Tao and Liang, Zhihong and Zhao, Jie},
  title = {Compiler-Guided Polynomial-Level Parallelism for FHE-Encrypted Machine Learning Inference},
  year = {2026},
  booktitle = {Proceedings of the 19th International Symposium on High-Level Parallel Programming and Applications},
  month = jul,
  location = {Paris, France},
  series = {HLPP 2026}
}

TACO 2026
A Decoupled Analytical Model for Tile Size Selection in Affine Programs

Shihan Yuan^*, Zuoyan Zhang^*, Guanghui Song, Junhui Peng, and 4 more authors

ACM Trans. Archit. Code Optim., Apr 2026

Abs DOI Bib PDF

Existing tile size selection approaches are tightly coupled with compiler transformation pipelines, often leading to inaccurate modeling of cache behavior and limited effectiveness for non-rectangular tile shapes. This paper presents TileMind, a decoupled analytical model that combines compile-time and runtime information for tile size selection in affine programs. It introduces a transformation-aware pre-tiling step that enables the decoupled selector to remain consistent with compiler transformations while extracting compile-time metadata. The extracted metadata is then combined with profiled runtime characteristics to construct a richer yet tractable feasible domain, within which a nonlinear objective for tile size selection is formulated. This objective is subsequently transformed into a binary product linearization problem, with its nonlinear constraints also linearized for efficient optimization. Finally, an intra-tile optimization aligns computation with data layout to enhance data reuse within tiles. Across two multi-core Intel CPUs, TileMind achieves 1.49x (sequential) and 1.33x (parallel) mean speedups on twenty PolyBench kernels, and 2.08-3.54x speedups on three deep learning workloads over the state-of-the-art analytical model Pluto-tss. Compared with TVM’s latest autotuner MetaSchedule, TileMind delivers 1.35-1.46x mean speedups while reducing tuning overhead by 2-4 orders of magnitude. While demonstrating effectiveness on selecting tile sizes for non-rectangular tile shapes and compatibility with PPCG, Pluto, and TVM, we further provide proof-of-concept results on GPUs, illustrating the potential portability of TileMind across architectures.
@article{tilemind2026, author = {Yuan, Shihan and Zhang, Zuoyan and Song, Guanghui and Peng, Junhui and Wang, Feng and Tang, Zhuo and Li, Kenli and Zhao, Jie}, title = {A Decoupled Analytical Model for Tile Size Selection in Affine Programs}, year = {2026}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, issn = {1544-3566}, url = {https://doi.org/10.1145/3806056}, doi = {10.1145/3806056}, journal = {ACM Trans. Archit. Code Optim.}, month = apr, keywords = {Loop Tiling, Polyhedral Compilation, Analytical Model, Cache Optimization} }

2025

QRS 2025
Scalable Detection of Floating-point Errors via Adaptive Parallel Subdomain Search

Zuoyan Zhang, Shihan Yuan, Hongru Yang, Jie Zhao, and 1 more author

In 2025 25th International Conference on Software Quality, Reliability and Security (QRS), Hangzhou, China, Apr 2025

Abs DOI Bib PDF Slides

Floating-point error detection is crucial in numerical computing, particularly for multi-parameter functions where even minor errors can propagate and significantly impact results. The sparse distribution of floating-point errors poses a significant detection challenge, as significant deviations are triggered by only rare inputs. Existing search algorithms face two major limitations: poor scalability for multi-parameter functions and insufficient utilization of floating-point representation characteristics. To address these challenges, we propose SDPS (Scalable Detection via Parallel Subdomain Search), a novel algorithm that combines adaptive domain partitioning with floating-point-specific heuristics. SDPS employs a multi-level error classification system and specialized point generation strategies, supported by efficient parallel processing through dynamic task allocation. Our comprehensive evaluation demonstrates that SDPS significantly outperforms state-of-the-art methods in both detection accuracy and computational efficiency, especially for multi-parameter functions where it effectively addresses the exponential growth of search space that limits existing approaches.
@inproceedings{qrs2025, author = {Zhang, Zuoyan and Yuan, Shihan and Yang, Hongru and Zhao, Jie and Xu, Jinchen}, title = {Scalable Detection of Floating-point Errors via Adaptive Parallel Subdomain Search}, year = {2025}, address = {Hangzhou, Zhejiang, China}, booktitle = {2025 25th International Conference on Software Quality, Reliability and Security (QRS)}, publisher = {IEEE}, pages = {392-403}, volume = {}, number = {}, keywords = {Software testing;Scalability;Software quality;Parallel processing;Partitioning algorithms;Classification algorithms;Software reliability;Performance analysis;Security;Resource management;floating-point error;error detection;program analysis;dynamic analysis;software testing}, location = {Hangzhou, China}, series = {QRS 2025}, doi = {10.1109/QRS65678.2025.00047} }

2024

ISSTA 2024
Arfa: An Agile Regime-Based Floating-Point Optimization Approach for Rounding Errors

Jinchen Xu^*, Mengqi Cui^*, Fei Li, Zuoyan Zhang, and 3 more authors

In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, Vienna, Austria, Apr 2024

Abs DOI Bib PDF Slides

We introduce a floating-point (FP) error optimization approach called Arfa that partitions the domain D of an FP expression fe into regimes and rewrites fe in each regime where fe shows larger errors. First, Arfa seeks a rewrite substitution fo with lower errors across D, whose error distribution is plotted for effective regime inference. Next, Arfa generates an incomplete set of ordered rewrite candidates within each regime of interest, so that searching for the best rewrite substitutions is performed efficiently. Finally, Arfa selects the best rewrite substitution by inspecting the errors of top ranked rewrite candidates, with enhancing precision also considered. Experiments on 56 FPbench examples and four real-life programs show that Arfa not only reduces the maximum and average errors of fe by 4.73 and 2.08 bits on average (and up to 33 and 16 bits), but also exhibits lower errors, sometimes to a significant degree, than Herbie and NumOpt.
@inproceedings{arfa, author = {Xu, Jinchen and Cui, Mengqi and Li, Fei and Zhang, Zuoyan and Yang, Hongru and Zhou, Bei and Zhao, Jie}, title = {Arfa: An Agile Regime-Based Floating-Point Optimization Approach for Rounding Errors}, year = {2024}, isbn = {9798400706127}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3650212.3680378}, doi = {10.1145/3650212.3680378}, booktitle = {Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis}, pages = {1516-1528}, numpages = {13}, keywords = {FPbench, dynamic analysis, e-graph, floating-point errors, numerical analysis, rewrite}, location = {Vienna, Austria}, series = {ISSTA 2024} }

2023

TJSC
Hierarchical search algorithm for error detection in floating-point arithmetic expressions

Zuoyan Zhang^*, Jinchen Xu^*, Jiangwei Hao, Yang Qu, and 2 more authors

J. Supercomput., Jul 2023

Abs DOI Bib PDF

Scientific and engineering applications rely on floating-point arithmetic to approximate real numbers. Due to the inherent rounding errors in floating-point numbers, error propagation during calculations can accumulate and lead to serious errors that may compromise the safety and reliability of the program. In theory, the most accurate method of error detection is to exhaustively search all possible floating-point inputs, but this is not feasible in practice due to the huge search space involved. Effectively and efficiently detecting maximum floating-point errors has been a challenge. To address this challenge, we design and implement an error detection tool for floating-point arithmetic expressions called HSED. It leverages modified mantissas under double precision floating-point types to simulate hierarchical searches from either half or single precision to double precision. Experimental results show that for 32 single-parameter arithmetic expressions in the FPBench benchmark test set, the error detection effects and performance of HSED are significantly better than the state-of-the-art error detection tools Herbie, S3FP and ATOMU. HSED outperforms Herbie, Herbie+, S3FP and ATOMU in 24, 19, 27 and 25 cases, respectively. The average time taken by Herbie, Herbie+, and S3FP is 1.82, 11.20, and 129.15 times longer than HSED, respectively.
@article{hsed, author = {Zhang, Zuoyan and Xu, Jinchen and Hao, Jiangwei and Qu, Yang and He, Haotian and Zhou, Bei}, title = {Hierarchical search algorithm for error detection in floating-point arithmetic expressions}, year = {2023}, issue_date = {Jan 2024}, publisher = {Kluwer Academic Publishers}, address = {USA}, volume = {80}, number = {1}, issn = {0920-8542}, url = {https://doi.org/10.1007/s11227-023-05523-6}, doi = {10.1007/s11227-023-05523-6}, journal = {J. Supercomput.}, month = jul, pages = {1183-1205}, numpages = {23}, keywords = {Floating-point arithmetic, Error detection, Dynamic analysis, Hierarchical search} }
ASE 2023
Eiffel: Inferring Input Ranges of Significant Floating-Point Errors via Polynomial Extrapolation

Zuoyan Zhang, Bei Zhou, Jiangwei Hao, Hongru Yang, and 6 more authors

In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, Echternach, Luxembourg, Jul 2023

Abs DOI Bib PDF Video Slides

Existing search heuristics used to find input values that result in significant floating-point (FP) errors or small ranges that cover them are accompanied by severe constraints, complicating their implementation and restricting their general applicability. This paper introduces an error analysis tool called Eiffel to infer error-inducing input ranges instead of searching them. Given an FP expression with its domain D, Eiffel first constructs an error data set by sampling values across a smaller domain ℛ and assembles these data into clusters. If more than two clusters are formed, Eiffel derives polynomial curves that best fit the bound coordinates of the error-inducing ranges in ℛ, extrapolating them to infer all target ranges of D and reporting the maximal error. Otherwise, Eiffel simply returns the largest error across ℛ. Experimental results show that Eiffel exhibits a broader applicability than Atomu and S3FP by successfully detecting the errors of all 70 considered benchmarks while the two baselines only report errors for part of them. By taking as input the inferred ranges of Eiffel, Herbie obtains an average accuracy improvement of 3.35 bits and up to 53.3 bits.
@inproceedings{ase 2023, author = {Zhang, Zuoyan and Zhou, Bei and Hao, Jiangwei and Yang, Hongru and Cui, Mengqi and Zhou, Yuchang and Song, Guanghui and Li, Fei and Xu, Jinchen and Zhao, Jie}, title = {Eiffel: Inferring Input Ranges of Significant Floating-Point Errors via Polynomial Extrapolation}, year = {2023}, isbn = {9798350329964}, publisher = {IEEE Press}, url = {https://doi.org/10.1109/ASE56229.2023.00139}, doi = {10.1109/ASE56229.2023.00139}, booktitle = {Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering}, pages = {1441-1453}, numpages = {13}, location = {Echternach, Luxembourg}, series = {ASE '23} }