Position: Mathematical DSL Benchmarks Need Executable Interface Contracts

Anna Genevaux, Simon Frieder
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:170262-170282, 2026.

Abstract

This position paper argues that documentation is infrastructure for reproducible geometry reasoning, and more broadly that mathematical benchmarks built around formal domain-specific languages (DSLs) need an executable interface contract: documented vocabulary should be matched by executable, versioned behavior and runnable examples. We use JGEX (as implemented by Newclid and AlphaGeometry) as a prototypical case study of how documentation–implementation gaps, missing examples, and nontrivial encoding choices can constrain expressivity, affect solver behavior, fragment tool interoperability, and bias benchmark construction. We introduce JGEXDiv, a curated collection of $137$ Euclidean geometry problems with released JGEX formalizations, natural-language rewrites, and metadata, covering $51.4%$ of admissible construction-pattern bins under our signature-based analysis. We also release a conformance suite for the documented JGEX interface, including minimal runnable examples, validation scripts, and a predicate-level support matrix. This audit surfaces cross-tool discrepancies, including documented predicates that execute under pinned Newclid v3.0.1 but are not recognized by Yuclid, as well as predicates that are unsupported or unstable under Newclid itself. Finally, we provide an executable NL$\rightarrow$JGEX baseline under the same backend. We argue that these artifacts together form an executable interface contract for mathematical DSL benchmarks, and should be treated as first-class research artifacts alongside datasets and evaluation code.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-genevaux26a, title = {Position: Mathematical {DSL} Benchmarks Need Executable Interface Contracts}, author = {Genevaux, Anna and Frieder, Simon}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {170262--170282}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/genevaux26a/genevaux26a.pdf}, url = {https://proceedings.mlr.press/v306/genevaux26a.html}, abstract = {This position paper argues that documentation is infrastructure for reproducible geometry reasoning, and more broadly that mathematical benchmarks built around formal domain-specific languages (DSLs) need an executable interface contract: documented vocabulary should be matched by executable, versioned behavior and runnable examples. We use JGEX (as implemented by Newclid and AlphaGeometry) as a prototypical case study of how documentation–implementation gaps, missing examples, and nontrivial encoding choices can constrain expressivity, affect solver behavior, fragment tool interoperability, and bias benchmark construction. We introduce JGEXDiv, a curated collection of $137$ Euclidean geometry problems with released JGEX formalizations, natural-language rewrites, and metadata, covering $51.4%$ of admissible construction-pattern bins under our signature-based analysis. We also release a conformance suite for the documented JGEX interface, including minimal runnable examples, validation scripts, and a predicate-level support matrix. This audit surfaces cross-tool discrepancies, including documented predicates that execute under pinned Newclid v3.0.1 but are not recognized by Yuclid, as well as predicates that are unsupported or unstable under Newclid itself. Finally, we provide an executable NL$\rightarrow$JGEX baseline under the same backend. We argue that these artifacts together form an executable interface contract for mathematical DSL benchmarks, and should be treated as first-class research artifacts alongside datasets and evaluation code.} }
Endnote
%0 Conference Paper %T Position: Mathematical DSL Benchmarks Need Executable Interface Contracts %A Anna Genevaux %A Simon Frieder %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-genevaux26a %I PMLR %P 170262--170282 %U https://proceedings.mlr.press/v306/genevaux26a.html %V 306 %X This position paper argues that documentation is infrastructure for reproducible geometry reasoning, and more broadly that mathematical benchmarks built around formal domain-specific languages (DSLs) need an executable interface contract: documented vocabulary should be matched by executable, versioned behavior and runnable examples. We use JGEX (as implemented by Newclid and AlphaGeometry) as a prototypical case study of how documentation–implementation gaps, missing examples, and nontrivial encoding choices can constrain expressivity, affect solver behavior, fragment tool interoperability, and bias benchmark construction. We introduce JGEXDiv, a curated collection of $137$ Euclidean geometry problems with released JGEX formalizations, natural-language rewrites, and metadata, covering $51.4%$ of admissible construction-pattern bins under our signature-based analysis. We also release a conformance suite for the documented JGEX interface, including minimal runnable examples, validation scripts, and a predicate-level support matrix. This audit surfaces cross-tool discrepancies, including documented predicates that execute under pinned Newclid v3.0.1 but are not recognized by Yuclid, as well as predicates that are unsupported or unstable under Newclid itself. Finally, we provide an executable NL$\rightarrow$JGEX baseline under the same backend. We argue that these artifacts together form an executable interface contract for mathematical DSL benchmarks, and should be treated as first-class research artifacts alongside datasets and evaluation code.
APA
Genevaux, A. & Frieder, S.. (2026). Position: Mathematical DSL Benchmarks Need Executable Interface Contracts. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:170262-170282 Available from https://proceedings.mlr.press/v306/genevaux26a.html.

Related Material