ContPhy: Continuum Physical Concept Learning and Reasoning from Videos

Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B. Tenenbaum, Chuang Gan
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:61526-61558, 2024.

Abstract

We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by encompassing the inference of diverse physical properties, such as mass and density, across various scenarios and predicting corresponding dynamics. We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset. We also introduce an oracle model (ContPRO) that marries the particle-based physical dynamic models with the recent large language models, which enjoy the advantages of both models, precise dynamic predictions, and interpretable reasoning. ContPhy aims to spur progress in perception and reasoning within diverse physical settings, narrowing the divide between human and machine intelligence in understanding the physical world.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zheng24l, title = {{C}ont{P}hy: Continuum Physical Concept Learning and Reasoning from Videos}, author = {Zheng, Zhicheng and Yan, Xin and Chen, Zhenfang and Wang, Jingzhou and Lim, Qin Zhi Eddie and Tenenbaum, Joshua B. and Gan, Chuang}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {61526--61558}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zheng24l/zheng24l.pdf}, url = {https://proceedings.mlr.press/v235/zheng24l.html}, abstract = {We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by encompassing the inference of diverse physical properties, such as mass and density, across various scenarios and predicting corresponding dynamics. We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset. We also introduce an oracle model (ContPRO) that marries the particle-based physical dynamic models with the recent large language models, which enjoy the advantages of both models, precise dynamic predictions, and interpretable reasoning. ContPhy aims to spur progress in perception and reasoning within diverse physical settings, narrowing the divide between human and machine intelligence in understanding the physical world.} }
Endnote
%0 Conference Paper %T ContPhy: Continuum Physical Concept Learning and Reasoning from Videos %A Zhicheng Zheng %A Xin Yan %A Zhenfang Chen %A Jingzhou Wang %A Qin Zhi Eddie Lim %A Joshua B. Tenenbaum %A Chuang Gan %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zheng24l %I PMLR %P 61526--61558 %U https://proceedings.mlr.press/v235/zheng24l.html %V 235 %X We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by encompassing the inference of diverse physical properties, such as mass and density, across various scenarios and predicting corresponding dynamics. We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset. We also introduce an oracle model (ContPRO) that marries the particle-based physical dynamic models with the recent large language models, which enjoy the advantages of both models, precise dynamic predictions, and interpretable reasoning. ContPhy aims to spur progress in perception and reasoning within diverse physical settings, narrowing the divide between human and machine intelligence in understanding the physical world.
APA
Zheng, Z., Yan, X., Chen, Z., Wang, J., Lim, Q.Z.E., Tenenbaum, J.B. & Gan, C.. (2024). ContPhy: Continuum Physical Concept Learning and Reasoning from Videos. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:61526-61558 Available from https://proceedings.mlr.press/v235/zheng24l.html.

Related Material