EPAS: Efficient Training with Progressive Activation Sharing

Rezaul Karim, Maryam Dialameh, Yang Liu, Boxing Chen, Walid Ahmed
Proceedings of the The 39th Canadian Conference on Artificial Intelligence, PMLR 318:1-12, 2026.

Abstract

We present a novel method for Efficient training with Progressive Activation Sharing (EPAS). This method bridges progressive training paradigm with the phenomenon of redundant representation across deeper layers of transformers. EPAS gradually grows an activation sharing region during training by switching decoder layers to activation sharing mode. This results in throughput increase due to reduced compute. To utilize deeper layer redundancy, the sharing region starts from the deep end of the model and grows towards the shallow end. The EPAS trained models allow for variable activation-sharing region lengths for different compute budgets during inference. Empirical evaluations with attention sharing (Q,K) in LLaMA models ranging from 125M to 7B parameters show up to an 11.1% improvement in training throughput and up to a 29% improvement in inference throughput while maintaining similar loss curve to the baseline models. Furthermore, applying EPAS in continual pretraining to transform TinyLLaMA into an attention-sharing model yields up to a 10% improvement in average accuracy over state-of-the-art methods, emphasizing the significance of progressive training in activation-sharing models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v318-karim26a, title = {EPAS: Efficient Training with Progressive Activation Sharing}, author = {Karim, Rezaul and Dialameh, Maryam and Liu, Yang and Chen, Boxing and Ahmed, Walid}, booktitle = {Proceedings of the The 39th Canadian Conference on Artificial Intelligence}, pages = {1--12}, year = {2026}, editor = {Bouzar-Benlabiod, Lydia and Leung, Carson}, volume = {318}, series = {Proceedings of Machine Learning Research}, month = {25--29 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v318/main/assets/karim26a/karim26a.pdf}, url = {https://proceedings.mlr.press/v318/karim26a.html}, abstract = {We present a novel method for Efficient training with Progressive Activation Sharing (EPAS). This method bridges progressive training paradigm with the phenomenon of redundant representation across deeper layers of transformers. EPAS gradually grows an activation sharing region during training by switching decoder layers to activation sharing mode. This results in throughput increase due to reduced compute. To utilize deeper layer redundancy, the sharing region starts from the deep end of the model and grows towards the shallow end. The EPAS trained models allow for variable activation-sharing region lengths for different compute budgets during inference. Empirical evaluations with attention sharing (Q,K) in LLaMA models ranging from 125M to 7B parameters show up to an 11.1% improvement in training throughput and up to a 29% improvement in inference throughput while maintaining similar loss curve to the baseline models. Furthermore, applying EPAS in continual pretraining to transform TinyLLaMA into an attention-sharing model yields up to a 10% improvement in average accuracy over state-of-the-art methods, emphasizing the significance of progressive training in activation-sharing models.} }
Endnote
%0 Conference Paper %T EPAS: Efficient Training with Progressive Activation Sharing %A Rezaul Karim %A Maryam Dialameh %A Yang Liu %A Boxing Chen %A Walid Ahmed %B Proceedings of the The 39th Canadian Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Lydia Bouzar-Benlabiod %E Carson Leung %F pmlr-v318-karim26a %I PMLR %P 1--12 %U https://proceedings.mlr.press/v318/karim26a.html %V 318 %X We present a novel method for Efficient training with Progressive Activation Sharing (EPAS). This method bridges progressive training paradigm with the phenomenon of redundant representation across deeper layers of transformers. EPAS gradually grows an activation sharing region during training by switching decoder layers to activation sharing mode. This results in throughput increase due to reduced compute. To utilize deeper layer redundancy, the sharing region starts from the deep end of the model and grows towards the shallow end. The EPAS trained models allow for variable activation-sharing region lengths for different compute budgets during inference. Empirical evaluations with attention sharing (Q,K) in LLaMA models ranging from 125M to 7B parameters show up to an 11.1% improvement in training throughput and up to a 29% improvement in inference throughput while maintaining similar loss curve to the baseline models. Furthermore, applying EPAS in continual pretraining to transform TinyLLaMA into an attention-sharing model yields up to a 10% improvement in average accuracy over state-of-the-art methods, emphasizing the significance of progressive training in activation-sharing models.
APA
Karim, R., Dialameh, M., Liu, Y., Chen, B. & Ahmed, W.. (2026). EPAS: Efficient Training with Progressive Activation Sharing. Proceedings of the The 39th Canadian Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 318:1-12 Available from https://proceedings.mlr.press/v318/karim26a.html.

Related Material