Neurosymbolic Learning on Activity Summarization of Video Data
Proceedings of the Third International Workshop on Self-Supervised Learning, PMLR 192:108-119, 2022.
Neurosymbolic learning systems have been shown to quickly discover how to interact with discrete representations of the world. Symbolic learners often allow for a higher level of understandability than neural networks which learn feature vectors for actions being taken, as seen in modern reinforcement learning systems. Symbolic learners excel at learning higher-level concepts, but struggle with certain types of generalization. Symbolic learners might benefit in such situations from a learned representation of the world. This paper discusses a pipeline that uses state-of-the-art object and pose detection neural networks as input to a symbolic learning system. We show how the knowledge from the symbolic system can automatically correct object and pose data from the neural network and hence provide corrected samples that can be used to incrementally train and improve the neural network. We show how symbolic learning techniques can improve action detection when given example ground truths by humans. We also demonstrate how novel actions that are not recognized by humans might be recognized by a learning engine capable of recognizing results and preconditions for an action to be valid.