Better Loss Landscape Visualization for Deep Neural Networks with Trajectory Information

Ruiqi Ding, Tao Li, Xiaolin Huang
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:311-326, 2024.

Abstract

The loss landscape of neural networks is a valuable perspective for studying the trainability, generalization, and robustness of networks, and hence its visualization has been extensively studied. Essentially, visualization methods project the parameter space into a low-dimensional subspace, resulting in a substantial loss of network parameter information. The key is to identify the direction of loss reduction in the visualized loss landscape. However, the existing methods generally focus on one simple point, make it challenging to properly capture the main properties of the landscape. An obvious and important problem is that regardless of whether the center point is the convergence or not, the current methods may depict it a local optimal point in the visualization. To address this issue, we propose a visualization method that relies on the whole training process not a single solution, better reflecting the actual training loss.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-ding24a, title = {Better Loss Landscape Visualization for Deep Neural Networks with Trajectory Information}, author = {Ding, Ruiqi and Li, Tao and Huang, Xiaolin}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {311--326}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/ding24a/ding24a.pdf}, url = {https://proceedings.mlr.press/v222/ding24a.html}, abstract = {The loss landscape of neural networks is a valuable perspective for studying the trainability, generalization, and robustness of networks, and hence its visualization has been extensively studied. Essentially, visualization methods project the parameter space into a low-dimensional subspace, resulting in a substantial loss of network parameter information. The key is to identify the direction of loss reduction in the visualized loss landscape. However, the existing methods generally focus on one simple point, make it challenging to properly capture the main properties of the landscape. An obvious and important problem is that regardless of whether the center point is the convergence or not, the current methods may depict it a local optimal point in the visualization. To address this issue, we propose a visualization method that relies on the whole training process not a single solution, better reflecting the actual training loss.} }
Endnote
%0 Conference Paper %T Better Loss Landscape Visualization for Deep Neural Networks with Trajectory Information %A Ruiqi Ding %A Tao Li %A Xiaolin Huang %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-ding24a %I PMLR %P 311--326 %U https://proceedings.mlr.press/v222/ding24a.html %V 222 %X The loss landscape of neural networks is a valuable perspective for studying the trainability, generalization, and robustness of networks, and hence its visualization has been extensively studied. Essentially, visualization methods project the parameter space into a low-dimensional subspace, resulting in a substantial loss of network parameter information. The key is to identify the direction of loss reduction in the visualized loss landscape. However, the existing methods generally focus on one simple point, make it challenging to properly capture the main properties of the landscape. An obvious and important problem is that regardless of whether the center point is the convergence or not, the current methods may depict it a local optimal point in the visualization. To address this issue, we propose a visualization method that relies on the whole training process not a single solution, better reflecting the actual training loss.
APA
Ding, R., Li, T. & Huang, X.. (2024). Better Loss Landscape Visualization for Deep Neural Networks with Trajectory Information. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:311-326 Available from https://proceedings.mlr.press/v222/ding24a.html.

Related Material