Confronting Domain Shift in Trained Neural Networks

Carianne Martinez, David A. Najera-Flores, Adam R. Brink, D. Dane Quinn, Eleni Chatzi, Stephanie Forrest
NeurIPS 2020 Workshop on Pre-registration in Machine Learning, PMLR 148:176-192, 2021.

Abstract

Neural networks (NNs) are known as universal function approximators and can interpolate nonlinear functions between observed data points. However, when the target domain for deployment shifts from the training domain and NNs must extrapolate, the results are notoriously poor. Prior work has shown that NN uncertainty estimates can be used to correct binary predictions in shifted domains without retraining the model. We hypothesize that this approach can be extended to correct real-valued time series predictions. As an exemplar, we consider two mechanical systems with nonlinear dynamics. The first system consists of a spring-mass system where the stiffness changes abruptly, and the second is a real experimental system with a frictional joint that is an open challenge for structural dynamicists to model efficiently. Our experiments will test whether 1) NN uncertainty estimates can identify when the input domain has shifted from the training domain and 2) whether the information used to calculate uncertainty estimates can be used to correct the NN’s time series predictions. While the method as proposed did not significantly improve predictions, our results did show potential for modifications that could improve models’ predictions and play a role in structural health monitoring systems that directly impact public safety.

Cite this Paper

BibTeX
@InProceedings{pmlr-v148-martinez21a, title = {Confronting Domain Shift in Trained Neural Networks}, author = {Martinez, Carianne and Najera-Flores, David A. and Brink, Adam R. and Quinn, D. Dane and Chatzi, Eleni and Forrest, Stephanie}, booktitle = {NeurIPS 2020 Workshop on Pre-registration in Machine Learning}, pages = {176--192}, year = {2021}, editor = {Bertinetto, Luca and Henriques, João F. and Albanie, Samuel and Paganini, Michela and Varol, Gül}, volume = {148}, series = {Proceedings of Machine Learning Research}, month = {11 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v148/martinez21a/martinez21a.pdf}, url = {https://proceedings.mlr.press/v148/martinez21a.html}, abstract = {Neural networks (NNs) are known as universal function approximators and can interpolate nonlinear functions between observed data points. However, when the target domain for deployment shifts from the training domain and NNs must extrapolate, the results are notoriously poor. Prior work has shown that NN uncertainty estimates can be used to correct binary predictions in shifted domains without retraining the model. We hypothesize that this approach can be extended to correct real-valued time series predictions. As an exemplar, we consider two mechanical systems with nonlinear dynamics. The first system consists of a spring-mass system where the stiffness changes abruptly, and the second is a real experimental system with a frictional joint that is an open challenge for structural dynamicists to model efficiently. Our experiments will test whether 1) NN uncertainty estimates can identify when the input domain has shifted from the training domain and 2) whether the information used to calculate uncertainty estimates can be used to correct the NN’s time series predictions. While the method as proposed did not significantly improve predictions, our results did show potential for modifications that could improve models’ predictions and play a role in structural health monitoring systems that directly impact public safety.} }
Endnote
%0 Conference Paper %T Confronting Domain Shift in Trained Neural Networks %A Carianne Martinez %A David A. Najera-Flores %A Adam R. Brink %A D. Dane Quinn %A Eleni Chatzi %A Stephanie Forrest %B NeurIPS 2020 Workshop on Pre-registration in Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Luca Bertinetto %E João F. Henriques %E Samuel Albanie %E Michela Paganini %E Gül Varol %F pmlr-v148-martinez21a %I PMLR %P 176--192 %U https://proceedings.mlr.press/v148/martinez21a.html %V 148 %X Neural networks (NNs) are known as universal function approximators and can interpolate nonlinear functions between observed data points. However, when the target domain for deployment shifts from the training domain and NNs must extrapolate, the results are notoriously poor. Prior work has shown that NN uncertainty estimates can be used to correct binary predictions in shifted domains without retraining the model. We hypothesize that this approach can be extended to correct real-valued time series predictions. As an exemplar, we consider two mechanical systems with nonlinear dynamics. The first system consists of a spring-mass system where the stiffness changes abruptly, and the second is a real experimental system with a frictional joint that is an open challenge for structural dynamicists to model efficiently. Our experiments will test whether 1) NN uncertainty estimates can identify when the input domain has shifted from the training domain and 2) whether the information used to calculate uncertainty estimates can be used to correct the NN’s time series predictions. While the method as proposed did not significantly improve predictions, our results did show potential for modifications that could improve models’ predictions and play a role in structural health monitoring systems that directly impact public safety.
APA
Martinez, C., Najera-Flores, D.A., Brink, A.R., Quinn, D.D., Chatzi, E. & Forrest, S.. (2021). Confronting Domain Shift in Trained Neural Networks. NeurIPS 2020 Workshop on Pre-registration in Machine Learning, in Proceedings of Machine Learning Research 148:176-192 Available from https://proceedings.mlr.press/v148/martinez21a.html.

Related Material