Feedback from Pixels: Output Regulation via Learning-based Scene View Synthesis

Murad Abu-Khalaf, Sertac Karaman, Daniela Rus
Proceedings of the 3rd Conference on Learning for Dynamics and Control, PMLR 144:828-841, 2021.

Abstract

We propose a novel controller synthesis involving feedback from pixels, whereby the measurement is a high dimensional signal representing a pixelated image with Red-Green-Blue (RGB) values. The approach neither requires feature extraction, nor object detection, nor visual correspondence. The control policy does not involve the estimation of states or similar latent representations. Instead, tracking is achieved directly in image space, with a model of the reference signal embedded as required by the internal model principle. The reference signal is generated by a neural network with learning-based scene view synthesis capabilities. Our approach does not require an end-to-end learning of a pixel-to-action control policy. The approach is applied to a motion control problem, namely the longitudinal dynamics of a car-following problem. We show how this approach lend itself to a tractable stability analysis with associated bounds critical to establishing trustworthiness and interpretability of the closed-loop dynamics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v144-abu-khalaf21a, title = {Feedback from Pixels: Output Regulation via Learning-based Scene View Synthesis}, author = {Abu-Khalaf, Murad and Karaman, Sertac and Rus, Daniela}, booktitle = {Proceedings of the 3rd Conference on Learning for Dynamics and Control}, pages = {828--841}, year = {2021}, editor = {Jadbabaie, Ali and Lygeros, John and Pappas, George J. and A. Parrilo, Pablo and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.}, volume = {144}, series = {Proceedings of Machine Learning Research}, month = {07 -- 08 June}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v144/abu-khalaf21a/abu-khalaf21a.pdf}, url = {https://proceedings.mlr.press/v144/abu-khalaf21a.html}, abstract = {We propose a novel controller synthesis involving feedback from pixels, whereby the measurement is a high dimensional signal representing a pixelated image with Red-Green-Blue (RGB) values. The approach neither requires feature extraction, nor object detection, nor visual correspondence. The control policy does not involve the estimation of states or similar latent representations. Instead, tracking is achieved directly in image space, with a model of the reference signal embedded as required by the internal model principle. The reference signal is generated by a neural network with learning-based scene view synthesis capabilities. Our approach does not require an end-to-end learning of a pixel-to-action control policy. The approach is applied to a motion control problem, namely the longitudinal dynamics of a car-following problem. We show how this approach lend itself to a tractable stability analysis with associated bounds critical to establishing trustworthiness and interpretability of the closed-loop dynamics.} }
Endnote
%0 Conference Paper %T Feedback from Pixels: Output Regulation via Learning-based Scene View Synthesis %A Murad Abu-Khalaf %A Sertac Karaman %A Daniela Rus %B Proceedings of the 3rd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2021 %E Ali Jadbabaie %E John Lygeros %E George J. Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire J. Tomlin %E Melanie N. Zeilinger %F pmlr-v144-abu-khalaf21a %I PMLR %P 828--841 %U https://proceedings.mlr.press/v144/abu-khalaf21a.html %V 144 %X We propose a novel controller synthesis involving feedback from pixels, whereby the measurement is a high dimensional signal representing a pixelated image with Red-Green-Blue (RGB) values. The approach neither requires feature extraction, nor object detection, nor visual correspondence. The control policy does not involve the estimation of states or similar latent representations. Instead, tracking is achieved directly in image space, with a model of the reference signal embedded as required by the internal model principle. The reference signal is generated by a neural network with learning-based scene view synthesis capabilities. Our approach does not require an end-to-end learning of a pixel-to-action control policy. The approach is applied to a motion control problem, namely the longitudinal dynamics of a car-following problem. We show how this approach lend itself to a tractable stability analysis with associated bounds critical to establishing trustworthiness and interpretability of the closed-loop dynamics.
APA
Abu-Khalaf, M., Karaman, S. & Rus, D.. (2021). Feedback from Pixels: Output Regulation via Learning-based Scene View Synthesis. Proceedings of the 3rd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 144:828-841 Available from https://proceedings.mlr.press/v144/abu-khalaf21a.html.

Related Material