Neurosymbolic Learning on Activity Summarization of Video Data

Steve Komrusch, Sanket Bhave, Mridul Banik, Henry Minsky
Proceedings of the Third International Workshop on Self-Supervised Learning, PMLR 192:108-119, 2022.

Abstract

Neurosymbolic learning systems have been shown to quickly discover how to interact with discrete representations of the world. Symbolic learners often allow for a higher level of understandability than neural networks which learn feature vectors for actions being taken, as seen in modern reinforcement learning systems. Symbolic learners excel at learning higher-level concepts, but struggle with certain types of generalization. Symbolic learners might benefit in such situations from a learned representation of the world. This paper discusses a pipeline that uses state-of-the-art object and pose detection neural networks as input to a symbolic learning system. We show how the knowledge from the symbolic system can automatically correct object and pose data from the neural network and hence provide corrected samples that can be used to incrementally train and improve the neural network. We show how symbolic learning techniques can improve action detection when given example ground truths by humans. We also demonstrate how novel actions that are not recognized by humans might be recognized by a learning engine capable of recognizing results and preconditions for an action to be valid.

Cite this Paper

BibTeX
@InProceedings{pmlr-v192-komrusch22b, title = {Neurosymbolic Learning on Activity Summarization of Video Data}, author = {Komrusch, Steve and Bhave, Sanket and Banik, Mridul and Minsky, Henry}, booktitle = {Proceedings of the Third International Workshop on Self-Supervised Learning}, pages = {108--119}, year = {2022}, editor = {Thórisson, Kristinn R.}, volume = {192}, series = {Proceedings of Machine Learning Research}, month = {28--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v192/komrusch22b/komrusch22b.pdf}, url = {https://proceedings.mlr.press/v192/komrusch22b.html}, abstract = {Neurosymbolic learning systems have been shown to quickly discover how to interact with discrete representations of the world. Symbolic learners often allow for a higher level of understandability than neural networks which learn feature vectors for actions being taken, as seen in modern reinforcement learning systems. Symbolic learners excel at learning higher-level concepts, but struggle with certain types of generalization. Symbolic learners might benefit in such situations from a learned representation of the world. This paper discusses a pipeline that uses state-of-the-art object and pose detection neural networks as input to a symbolic learning system. We show how the knowledge from the symbolic system can automatically correct object and pose data from the neural network and hence provide corrected samples that can be used to incrementally train and improve the neural network. We show how symbolic learning techniques can improve action detection when given example ground truths by humans. We also demonstrate how novel actions that are not recognized by humans might be recognized by a learning engine capable of recognizing results and preconditions for an action to be valid.} }
Endnote
%0 Conference Paper %T Neurosymbolic Learning on Activity Summarization of Video Data %A Steve Komrusch %A Sanket Bhave %A Mridul Banik %A Henry Minsky %B Proceedings of the Third International Workshop on Self-Supervised Learning %C Proceedings of Machine Learning Research %D 2022 %E Kristinn R. Thórisson %F pmlr-v192-komrusch22b %I PMLR %P 108--119 %U https://proceedings.mlr.press/v192/komrusch22b.html %V 192 %X Neurosymbolic learning systems have been shown to quickly discover how to interact with discrete representations of the world. Symbolic learners often allow for a higher level of understandability than neural networks which learn feature vectors for actions being taken, as seen in modern reinforcement learning systems. Symbolic learners excel at learning higher-level concepts, but struggle with certain types of generalization. Symbolic learners might benefit in such situations from a learned representation of the world. This paper discusses a pipeline that uses state-of-the-art object and pose detection neural networks as input to a symbolic learning system. We show how the knowledge from the symbolic system can automatically correct object and pose data from the neural network and hence provide corrected samples that can be used to incrementally train and improve the neural network. We show how symbolic learning techniques can improve action detection when given example ground truths by humans. We also demonstrate how novel actions that are not recognized by humans might be recognized by a learning engine capable of recognizing results and preconditions for an action to be valid.
APA
Komrusch, S., Bhave, S., Banik, M. & Minsky, H.. (2022). Neurosymbolic Learning on Activity Summarization of Video Data. Proceedings of the Third International Workshop on Self-Supervised Learning, in Proceedings of Machine Learning Research 192:108-119 Available from https://proceedings.mlr.press/v192/komrusch22b.html.

Related Material