Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"

Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, Kazuhito Koishida
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:279-290, 2020.

Abstract

Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a Differentiable First-Order Logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-amizadeh20a, title = {Neuro-Symbolic Visual Reasoning: Disentangling "{V}isual" from "{R}easoning"}, author = {Amizadeh, Saeed and Palangi, Hamid and Polozov, Alex and Huang, Yichen and Koishida, Kazuhito}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {279--290}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/amizadeh20a/amizadeh20a.pdf}, url = {https://proceedings.mlr.press/v119/amizadeh20a.html}, abstract = {Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a Differentiable First-Order Logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.} }
Endnote
%0 Conference Paper %T Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning" %A Saeed Amizadeh %A Hamid Palangi %A Alex Polozov %A Yichen Huang %A Kazuhito Koishida %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-amizadeh20a %I PMLR %P 279--290 %U https://proceedings.mlr.press/v119/amizadeh20a.html %V 119 %X Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. To this end, we introduce a Differentiable First-Order Logic formalism for VQA that explicitly decouples question answering from visual perception. On the challenging GQA dataset, this framework is used to perform in-depth, disentangled comparisons between well-known VQA models leading to informative insights regarding the participating models as well as the task.
APA
Amizadeh, S., Palangi, H., Polozov, A., Huang, Y. & Koishida, K.. (2020). Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning". Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:279-290 Available from https://proceedings.mlr.press/v119/amizadeh20a.html.

Related Material