Vision-based System Identification and 3D Keypoint Discovery using Dynamics Constraints

Miguel Jaques, Martin Asenov, Michael Burke, Timothy Hospedales
Proceedings of The 4th Annual Learning for Dynamics and Control Conference, PMLR 168:316-329, 2022.

Abstract

This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v168-jaques22a, title = {Vision-based System Identification and 3D Keypoint Discovery using Dynamics Constraints}, author = {Jaques, Miguel and Asenov, Martin and Burke, Michael and Hospedales, Timothy}, booktitle = {Proceedings of The 4th Annual Learning for Dynamics and Control Conference}, pages = {316--329}, year = {2022}, editor = {Firoozi, Roya and Mehr, Negar and Yel, Esen and Antonova, Rika and Bohg, Jeannette and Schwager, Mac and Kochenderfer, Mykel}, volume = {168}, series = {Proceedings of Machine Learning Research}, month = {23--24 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v168/jaques22a/jaques22a.pdf}, url = {https://proceedings.mlr.press/v168/jaques22a.html}, abstract = {This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.} }
Endnote
%0 Conference Paper %T Vision-based System Identification and 3D Keypoint Discovery using Dynamics Constraints %A Miguel Jaques %A Martin Asenov %A Michael Burke %A Timothy Hospedales %B Proceedings of The 4th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2022 %E Roya Firoozi %E Negar Mehr %E Esen Yel %E Rika Antonova %E Jeannette Bohg %E Mac Schwager %E Mykel Kochenderfer %F pmlr-v168-jaques22a %I PMLR %P 316--329 %U https://proceedings.mlr.press/v168/jaques22a.html %V 168 %X This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.
APA
Jaques, M., Asenov, M., Burke, M. & Hospedales, T.. (2022). Vision-based System Identification and 3D Keypoint Discovery using Dynamics Constraints. Proceedings of The 4th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 168:316-329 Available from https://proceedings.mlr.press/v168/jaques22a.html.

Related Material