Uncovering Hidden Training Dynamics in Neural Networks via Inter-Sample Influence Graphs

Dylan Tan Hong Tai, Jiayin Zhang, Rohan Ghosh, Mehul Motani
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:4807-4815, 2026.

Abstract

Deep learning models are primarily trained through batchwise optimization, where each update can potentially be a tug-of-war among samples, shaping the overall trajectory of learning. Existing interpretability tools, most notably influence functions, have provided valuable insights into how individual training samples affect model predictions, primarily at test time. However, these methods were not intended to capture these inter-sample interactions that arise during training. Here, we ask a complementary question: How does optimizing the loss on one training sample affect the loss on the rest during learning? We introduce Influence Graphs (IGs), directed inter-sample graphs where each edge weight $w_{ij}$ quantifies how optimizing on sample $X_i$ influences the loss of sample $X_j$. We estimate these influences via simulated batch interventions and slope coefficients of loss changes, enabling scalable construction of IGs during training. We further define the Mean-of-Mean In-Degree Influence (MMDI) and prove it bounds generalization under practical assumptions. Empirically, MMDI correlates strongly with test accuracy in noisy-label settings, making it a useful diagnostic of model quality even before test metrics are available. Finally, we show that IGs reveal distinct, evolving training phases, offering a new lens on the dynamics of learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-tai26a, title = { Uncovering Hidden Training Dynamics in Neural Networks via Inter-Sample Influence Graphs }, author = {Tai, Dylan Tan Hong and Zhang, Jiayin and Ghosh, Rohan and Motani, Mehul}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {4807--4815}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/tai26a/tai26a.pdf}, url = {https://proceedings.mlr.press/v300/tai26a.html}, abstract = { Deep learning models are primarily trained through batchwise optimization, where each update can potentially be a tug-of-war among samples, shaping the overall trajectory of learning. Existing interpretability tools, most notably influence functions, have provided valuable insights into how individual training samples affect model predictions, primarily at test time. However, these methods were not intended to capture these inter-sample interactions that arise during training. Here, we ask a complementary question: How does optimizing the loss on one training sample affect the loss on the rest during learning? We introduce Influence Graphs (IGs), directed inter-sample graphs where each edge weight $w_{ij}$ quantifies how optimizing on sample $X_i$ influences the loss of sample $X_j$. We estimate these influences via simulated batch interventions and slope coefficients of loss changes, enabling scalable construction of IGs during training. We further define the Mean-of-Mean In-Degree Influence (MMDI) and prove it bounds generalization under practical assumptions. Empirically, MMDI correlates strongly with test accuracy in noisy-label settings, making it a useful diagnostic of model quality even before test metrics are available. Finally, we show that IGs reveal distinct, evolving training phases, offering a new lens on the dynamics of learning. } }
Endnote
%0 Conference Paper %T Uncovering Hidden Training Dynamics in Neural Networks via Inter-Sample Influence Graphs %A Dylan Tan Hong Tai %A Jiayin Zhang %A Rohan Ghosh %A Mehul Motani %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-tai26a %I PMLR %P 4807--4815 %U https://proceedings.mlr.press/v300/tai26a.html %V 300 %X Deep learning models are primarily trained through batchwise optimization, where each update can potentially be a tug-of-war among samples, shaping the overall trajectory of learning. Existing interpretability tools, most notably influence functions, have provided valuable insights into how individual training samples affect model predictions, primarily at test time. However, these methods were not intended to capture these inter-sample interactions that arise during training. Here, we ask a complementary question: How does optimizing the loss on one training sample affect the loss on the rest during learning? We introduce Influence Graphs (IGs), directed inter-sample graphs where each edge weight $w_{ij}$ quantifies how optimizing on sample $X_i$ influences the loss of sample $X_j$. We estimate these influences via simulated batch interventions and slope coefficients of loss changes, enabling scalable construction of IGs during training. We further define the Mean-of-Mean In-Degree Influence (MMDI) and prove it bounds generalization under practical assumptions. Empirically, MMDI correlates strongly with test accuracy in noisy-label settings, making it a useful diagnostic of model quality even before test metrics are available. Finally, we show that IGs reveal distinct, evolving training phases, offering a new lens on the dynamics of learning.
APA
Tai, D.T.H., Zhang, J., Ghosh, R. & Motani, M.. (2026). Uncovering Hidden Training Dynamics in Neural Networks via Inter-Sample Influence Graphs . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:4807-4815 Available from https://proceedings.mlr.press/v300/tai26a.html.

Related Material