FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

Fedor Rodionov, Abdelrahman Eldesokey, Michael Birsak, John Femiani, Bernard Ghanem, Peter Wonka
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:105474-105511, 2026.

Abstract

We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed on shallow queries, they often fail to respect physical constraints and preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today’s LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-rodionov26a, title = {{F}loorplan{QA}: A Benchmark for Spatial Reasoning in {LLM}s using Structured Representations}, author = {Rodionov, Fedor and Eldesokey, Abdelrahman and Birsak, Michael and Femiani, John and Ghanem, Bernard and Wonka, Peter}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {105474--105511}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/rodionov26a/rodionov26a.pdf}, url = {https://proceedings.mlr.press/v306/rodionov26a.html}, abstract = {We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed on shallow queries, they often fail to respect physical constraints and preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today’s LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings.} }
Endnote
%0 Conference Paper %T FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations %A Fedor Rodionov %A Abdelrahman Eldesokey %A Michael Birsak %A John Femiani %A Bernard Ghanem %A Peter Wonka %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-rodionov26a %I PMLR %P 105474--105511 %U https://proceedings.mlr.press/v306/rodionov26a.html %V 306 %X We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed on shallow queries, they often fail to respect physical constraints and preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today’s LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings.
APA
Rodionov, F., Eldesokey, A., Birsak, M., Femiani, J., Ghanem, B. & Wonka, P.. (2026). FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:105474-105511 Available from https://proceedings.mlr.press/v306/rodionov26a.html.

Related Material