Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback

Natasha Jaques, Jennifer McCleary, Jesse Engel, David Ha, Fred Bertsch, Douglas Eck, Rosalind Picard
Proceedings of IJCAI 2018 2nd Workshop on Artificial Intelligence in Affective Computing, PMLR 86:1-9, 2020.

Abstract

A known deficit of modern machine learning (ML) and deep learning (DL) methodology is that models must be carefully fine-tuned in order to solve a particular task. Most algorithms cannot generalize well to even highly similar tasks, let alone exhibit signs of artificial general intelligence (AGI). To address this problem, researchers have explored developing loss functions that act as intrinsic motivators that could drive an ML or DL agent to learn across a number of domains. This paper argues that an important and useful intrinsic motivator is that of social interaction. We posit that making an AI agent aware of implicit social feedback from humans can allow for faster learning of more generalizable and useful representations, and could potentially impact AI safety. We collect social feedback in the form of facial expression reactions to samples from Sketch RNN, an LSTM-based variational autoencoder (VAE) designed to produce sketch drawings. We use a Latent Constraints GAN (LC-GAN) to learn from the facial feedback of a small group of viewers, by optimizing the model to produce sketches that it predicts will lead to more positive facial expressions. We show in multiple independent evaluations that the model trained with facial feedback produced sketches that are more highly rated, and induce significantly more positive facial expressions. Thus, we establish that implicit social feedback can improve the output of a deep learning model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v86-jaques20a, title = {Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback}, author = {Jaques, Natasha and McCleary, Jennifer and Engel, Jesse and Ha, David and Bertsch, Fred and Eck, Douglas and Picard, Rosalind}, booktitle = {Proceedings of IJCAI 2018 2nd Workshop on Artificial Intelligence in Affective Computing}, pages = {1--9}, year = {2020}, editor = {Hsu, William and Yates, Heath}, volume = {86}, series = {Proceedings of Machine Learning Research}, month = {15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v86/jaques20a/jaques20a.pdf}, url = {http://proceedings.mlr.press/v86/jaques20a.html}, abstract = {A known deficit of modern machine learning (ML) and deep learning (DL) methodology is that models must be carefully fine-tuned in order to solve a particular task. Most algorithms cannot generalize well to even highly similar tasks, let alone exhibit signs of artificial general intelligence (AGI). To address this problem, researchers have explored developing loss functions that act as intrinsic motivators that could drive an ML or DL agent to learn across a number of domains. This paper argues that an important and useful intrinsic motivator is that of social interaction. We posit that making an AI agent aware of implicit social feedback from humans can allow for faster learning of more generalizable and useful representations, and could potentially impact AI safety. We collect social feedback in the form of facial expression reactions to samples from Sketch RNN, an LSTM-based variational autoencoder (VAE) designed to produce sketch drawings. We use a Latent Constraints GAN (LC-GAN) to learn from the facial feedback of a small group of viewers, by optimizing the model to produce sketches that it predicts will lead to more positive facial expressions. We show in multiple independent evaluations that the model trained with facial feedback produced sketches that are more highly rated, and induce significantly more positive facial expressions. Thus, we establish that implicit social feedback can improve the output of a deep learning model.} }
Endnote
%0 Conference Paper %T Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback %A Natasha Jaques %A Jennifer McCleary %A Jesse Engel %A David Ha %A Fred Bertsch %A Douglas Eck %A Rosalind Picard %B Proceedings of IJCAI 2018 2nd Workshop on Artificial Intelligence in Affective Computing %C Proceedings of Machine Learning Research %D 2020 %E William Hsu %E Heath Yates %F pmlr-v86-jaques20a %I PMLR %P 1--9 %U http://proceedings.mlr.press/v86/jaques20a.html %V 86 %X A known deficit of modern machine learning (ML) and deep learning (DL) methodology is that models must be carefully fine-tuned in order to solve a particular task. Most algorithms cannot generalize well to even highly similar tasks, let alone exhibit signs of artificial general intelligence (AGI). To address this problem, researchers have explored developing loss functions that act as intrinsic motivators that could drive an ML or DL agent to learn across a number of domains. This paper argues that an important and useful intrinsic motivator is that of social interaction. We posit that making an AI agent aware of implicit social feedback from humans can allow for faster learning of more generalizable and useful representations, and could potentially impact AI safety. We collect social feedback in the form of facial expression reactions to samples from Sketch RNN, an LSTM-based variational autoencoder (VAE) designed to produce sketch drawings. We use a Latent Constraints GAN (LC-GAN) to learn from the facial feedback of a small group of viewers, by optimizing the model to produce sketches that it predicts will lead to more positive facial expressions. We show in multiple independent evaluations that the model trained with facial feedback produced sketches that are more highly rated, and induce significantly more positive facial expressions. Thus, we establish that implicit social feedback can improve the output of a deep learning model.
APA
Jaques, N., McCleary, J., Engel, J., Ha, D., Bertsch, F., Eck, D. & Picard, R.. (2020). Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback. Proceedings of IJCAI 2018 2nd Workshop on Artificial Intelligence in Affective Computing, in Proceedings of Machine Learning Research 86:1-9 Available from http://proceedings.mlr.press/v86/jaques20a.html.

Related Material