Expert Collapse and Compositional Failure in Simple Multimodal MoE

Anthony Ticinovic, Caren Han
Proceedings of AAAI 2026 Workshop on Bias in Multimodal AI, PMLR 332:1-10, 2026.

Abstract

Mixture-of-Experts (MoE) is a technique that uses multiple MLPs at each transformer layer, rather than using a single MLP. MoE architectures are hypothesised to create specialised experts, but this is often inferred rather than quantitatively measured. Further, this specialisation is in conflict with standard load-balancing losses that promote more uniform load distribution, though can encourage expert redundancy. We construct a novel two-unimodal-expert (vision/text) MoE testbed and use a three-stage protocol to first force specialisation with uni-modal data (Stage 2), then test its stability during fine-tuning (Stage 3). Our findings demonstrate that while Stage 2 successfully creates specialised experts, this specialisation persists only at the object-level. In Stage 3, the standard multimodal loss actively overwrites this structure, causing the latent space to default to only clustering by modality, rather than by concept. We identify the mechanism as layer-level expert collapse. Furthermore, a case study on compositional binding reveals that even when specialisation is present, it captures monolithic objects (e.g., ’car’) but fails to bind attributes like colour, perhaps highlighting a source of bias in multimodal representation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v332-ticinovic26a, title = {Expert Collapse and Compositional Failure in Simple Multimodal MoE}, author = {Ticinovic, Anthony and Han, Caren}, booktitle = {Proceedings of AAAI 2026 Workshop on Bias in Multimodal AI}, pages = {1--10}, year = {2026}, editor = {Han, Soyeon Caren and Cabral, Rina Carines}, volume = {332}, series = {Proceedings of Machine Learning Research}, month = {25 Jan}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v332/main/assets/ticinovic26a/ticinovic26a.pdf}, url = {https://proceedings.mlr.press/v332/ticinovic26a.html}, abstract = {Mixture-of-Experts (MoE) is a technique that uses multiple MLPs at each transformer layer, rather than using a single MLP. MoE architectures are hypothesised to create specialised experts, but this is often inferred rather than quantitatively measured. Further, this specialisation is in conflict with standard load-balancing losses that promote more uniform load distribution, though can encourage expert redundancy. We construct a novel two-unimodal-expert (vision/text) MoE testbed and use a three-stage protocol to first force specialisation with uni-modal data (Stage 2), then test its stability during fine-tuning (Stage 3). Our findings demonstrate that while Stage 2 successfully creates specialised experts, this specialisation persists only at the object-level. In Stage 3, the standard multimodal loss actively overwrites this structure, causing the latent space to default to only clustering by modality, rather than by concept. We identify the mechanism as layer-level expert collapse. Furthermore, a case study on compositional binding reveals that even when specialisation is present, it captures monolithic objects (e.g., ’car’) but fails to bind attributes like colour, perhaps highlighting a source of bias in multimodal representation.} }
Endnote
%0 Conference Paper %T Expert Collapse and Compositional Failure in Simple Multimodal MoE %A Anthony Ticinovic %A Caren Han %B Proceedings of AAAI 2026 Workshop on Bias in Multimodal AI %C Proceedings of Machine Learning Research %D 2026 %E Soyeon Caren Han %E Rina Carines Cabral %F pmlr-v332-ticinovic26a %I PMLR %P 1--10 %U https://proceedings.mlr.press/v332/ticinovic26a.html %V 332 %X Mixture-of-Experts (MoE) is a technique that uses multiple MLPs at each transformer layer, rather than using a single MLP. MoE architectures are hypothesised to create specialised experts, but this is often inferred rather than quantitatively measured. Further, this specialisation is in conflict with standard load-balancing losses that promote more uniform load distribution, though can encourage expert redundancy. We construct a novel two-unimodal-expert (vision/text) MoE testbed and use a three-stage protocol to first force specialisation with uni-modal data (Stage 2), then test its stability during fine-tuning (Stage 3). Our findings demonstrate that while Stage 2 successfully creates specialised experts, this specialisation persists only at the object-level. In Stage 3, the standard multimodal loss actively overwrites this structure, causing the latent space to default to only clustering by modality, rather than by concept. We identify the mechanism as layer-level expert collapse. Furthermore, a case study on compositional binding reveals that even when specialisation is present, it captures monolithic objects (e.g., ’car’) but fails to bind attributes like colour, perhaps highlighting a source of bias in multimodal representation.
APA
Ticinovic, A. & Han, C.. (2026). Expert Collapse and Compositional Failure in Simple Multimodal MoE. Proceedings of AAAI 2026 Workshop on Bias in Multimodal AI, in Proceedings of Machine Learning Research 332:1-10 Available from https://proceedings.mlr.press/v332/ticinovic26a.html.

Related Material