SCHUR-A*: Layer-wise Optimal Expert Pruning for MoEs via Schur-Complement Guided A* Search

Zheng Chen, Yang Weifeng, Jianxiao Tang, Buhui Yao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:17371-17386, 2026.

Abstract

Sparse Mixture-of-Experts (MoE) language models enable conditional computation but face deployment challenges due to the memory wall: while few experts are activated per token, the entire model must reside in memory. Existing expert pruning methods primarily rely on independent ranking, failing to account for the complex inter-dependencies and redundancies between experts. In this paper, we formulate post-training MoE pruning as a reconstruction-driven subset selection problem, aiming to minimize layer-output distortion under a cardinality constraint. We introduce SCHUR-A, an algorithm that leverages A search to achieve globally optimal expert selection within each layer. To maintain computational tractability, we derive a novel, admissible heuristic upper bound using a Schur-complement-based relaxation of the reconstruction objective. This tight bound allows for aggressive pruning of the search space while mathematically guaranteeing optimality. Furthermore, we propose an automated strategy to balance fidelity and memory reduction across heterogeneous layers via knee-point detection. Extensive experiments on Qwen3-30B-A3B demonstrate that SCHUR-A* significantly outperforms greedy and ranking-based baselines, maintaining comparable performance even under aggressive pruning ratios.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26ew, title = {{SCHUR}-A*: Layer-wise Optimal Expert Pruning for {M}o{E}s via Schur-Complement Guided A* Search}, author = {Chen, Zheng and Weifeng, Yang and Tang, Jianxiao and Yao, Buhui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {17371--17386}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26ew/chen26ew.pdf}, url = {https://proceedings.mlr.press/v306/chen26ew.html}, abstract = {Sparse Mixture-of-Experts (MoE) language models enable conditional computation but face deployment challenges due to the memory wall: while few experts are activated per token, the entire model must reside in memory. Existing expert pruning methods primarily rely on independent ranking, failing to account for the complex inter-dependencies and redundancies between experts. In this paper, we formulate post-training MoE pruning as a reconstruction-driven subset selection problem, aiming to minimize layer-output distortion under a cardinality constraint. We introduce SCHUR-A, an algorithm that leverages A search to achieve globally optimal expert selection within each layer. To maintain computational tractability, we derive a novel, admissible heuristic upper bound using a Schur-complement-based relaxation of the reconstruction objective. This tight bound allows for aggressive pruning of the search space while mathematically guaranteeing optimality. Furthermore, we propose an automated strategy to balance fidelity and memory reduction across heterogeneous layers via knee-point detection. Extensive experiments on Qwen3-30B-A3B demonstrate that SCHUR-A* significantly outperforms greedy and ranking-based baselines, maintaining comparable performance even under aggressive pruning ratios.} }
Endnote
%0 Conference Paper %T SCHUR-A*: Layer-wise Optimal Expert Pruning for MoEs via Schur-Complement Guided A* Search %A Zheng Chen %A Yang Weifeng %A Jianxiao Tang %A Buhui Yao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26ew %I PMLR %P 17371--17386 %U https://proceedings.mlr.press/v306/chen26ew.html %V 306 %X Sparse Mixture-of-Experts (MoE) language models enable conditional computation but face deployment challenges due to the memory wall: while few experts are activated per token, the entire model must reside in memory. Existing expert pruning methods primarily rely on independent ranking, failing to account for the complex inter-dependencies and redundancies between experts. In this paper, we formulate post-training MoE pruning as a reconstruction-driven subset selection problem, aiming to minimize layer-output distortion under a cardinality constraint. We introduce SCHUR-A, an algorithm that leverages A search to achieve globally optimal expert selection within each layer. To maintain computational tractability, we derive a novel, admissible heuristic upper bound using a Schur-complement-based relaxation of the reconstruction objective. This tight bound allows for aggressive pruning of the search space while mathematically guaranteeing optimality. Furthermore, we propose an automated strategy to balance fidelity and memory reduction across heterogeneous layers via knee-point detection. Extensive experiments on Qwen3-30B-A3B demonstrate that SCHUR-A* significantly outperforms greedy and ranking-based baselines, maintaining comparable performance even under aggressive pruning ratios.
APA
Chen, Z., Weifeng, Y., Tang, J. & Yao, B.. (2026). SCHUR-A*: Layer-wise Optimal Expert Pruning for MoEs via Schur-Complement Guided A* Search. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:17371-17386 Available from https://proceedings.mlr.press/v306/chen26ew.html.

Related Material