Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation

Haoran Zhang, Yafu Li, Xuyang Hu, Dongrui Liu, Zhilin Wang, Bo Li, Yu Cheng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:156437-156508, 2026.

Abstract

Large language models (LLMs) are increasingly applied in diverse real-world applications, each governed by bespoke behavioral and safety specifications (spec) custom-tailored by users or organizations. These specifications, categorized into safety-spec and behavioral-spec, vary across scenarios and evolve with changing preferences and requirements. We formalize this challenge as specification alignment, focusing on LLMs’ ability to follow dynamic, scenario-specific spec from both behavioral and safety perspectives. To address this challenge, we introduce SpecBench, a unified benchmark for measuring specification alignment, covering 5 scenarios, 103 spec, and 1,500 prompts. Experiments on 15 reasoning and 18 instruct models with several Test-Time Deliberation (TTD) methods, including Self-Refine, TPO, and MoreThink, show that SpecBench effectively reveals alignment gaps and that test-time deliberation improves specification alignment. Based on previous TTD methods, we further propose Align3, a lightweight method with hierarchical reflection and revision to reason over specification boundaries, advancing the safety-helpfulness trade-off frontier with minimal overhead. These results highlight test-time deliberation as an effective strategy for reasoning over the real-world specification boundaries. Our code and resources are available at https://github.com/zzzhr97/SpecBench.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26cl, title = {Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation}, author = {Zhang, Haoran and Li, Yafu and Hu, Xuyang and Liu, Dongrui and Wang, Zhilin and Li, Bo and Cheng, Yu}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {156437--156508}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26cl/zhang26cl.pdf}, url = {https://proceedings.mlr.press/v306/zhang26cl.html}, abstract = {Large language models (LLMs) are increasingly applied in diverse real-world applications, each governed by bespoke behavioral and safety specifications (spec) custom-tailored by users or organizations. These specifications, categorized into safety-spec and behavioral-spec, vary across scenarios and evolve with changing preferences and requirements. We formalize this challenge as specification alignment, focusing on LLMs’ ability to follow dynamic, scenario-specific spec from both behavioral and safety perspectives. To address this challenge, we introduce SpecBench, a unified benchmark for measuring specification alignment, covering 5 scenarios, 103 spec, and 1,500 prompts. Experiments on 15 reasoning and 18 instruct models with several Test-Time Deliberation (TTD) methods, including Self-Refine, TPO, and MoreThink, show that SpecBench effectively reveals alignment gaps and that test-time deliberation improves specification alignment. Based on previous TTD methods, we further propose Align3, a lightweight method with hierarchical reflection and revision to reason over specification boundaries, advancing the safety-helpfulness trade-off frontier with minimal overhead. These results highlight test-time deliberation as an effective strategy for reasoning over the real-world specification boundaries. Our code and resources are available at https://github.com/zzzhr97/SpecBench.} }
Endnote
%0 Conference Paper %T Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation %A Haoran Zhang %A Yafu Li %A Xuyang Hu %A Dongrui Liu %A Zhilin Wang %A Bo Li %A Yu Cheng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26cl %I PMLR %P 156437--156508 %U https://proceedings.mlr.press/v306/zhang26cl.html %V 306 %X Large language models (LLMs) are increasingly applied in diverse real-world applications, each governed by bespoke behavioral and safety specifications (spec) custom-tailored by users or organizations. These specifications, categorized into safety-spec and behavioral-spec, vary across scenarios and evolve with changing preferences and requirements. We formalize this challenge as specification alignment, focusing on LLMs’ ability to follow dynamic, scenario-specific spec from both behavioral and safety perspectives. To address this challenge, we introduce SpecBench, a unified benchmark for measuring specification alignment, covering 5 scenarios, 103 spec, and 1,500 prompts. Experiments on 15 reasoning and 18 instruct models with several Test-Time Deliberation (TTD) methods, including Self-Refine, TPO, and MoreThink, show that SpecBench effectively reveals alignment gaps and that test-time deliberation improves specification alignment. Based on previous TTD methods, we further propose Align3, a lightweight method with hierarchical reflection and revision to reason over specification boundaries, advancing the safety-helpfulness trade-off frontier with minimal overhead. These results highlight test-time deliberation as an effective strategy for reasoning over the real-world specification boundaries. Our code and resources are available at https://github.com/zzzhr97/SpecBench.
APA
Zhang, H., Li, Y., Hu, X., Liu, D., Wang, Z., Li, B. & Cheng, Y.. (2026). Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:156437-156508 Available from https://proceedings.mlr.press/v306/zhang26cl.html.

Related Material