Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalization

Foundational to interpreting pretrained representations of deep generative models, the linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, linear representation does not imply linear accessibility of such concepts: under superposition, when the number of concepts exceeds the activation dimension, recovering the underlying latent factors requires sparse nonlinear inference, making methods such as linear probes insufficient. Sparse autoencoders ({SAEs}) perform nonlinear inference but amortize it into a fixed encoder, introducing a systematic amortization gap. We show this gap dominates all other error sources and persists as the number of training samples is increased, causing {SAEs} to fail under out-of-distribution ({OOD}) compositional shifts. In contrast, classical sparse coding with per-sample iterative inference leverages compressed sensing guarantees to recover latent factors robustly, maintaining near-zero gaps in the accuracy between in and out of distribution. Our results demonstrate that the recent {OOD} failures of {SAEs} can be attributed to amortization failures: per-sample inference at test time substantially improves {OOD} performance, even when using a dictionary learned by an {SAE}. This is observed along a spectrum of hybrid approaches that progressively undo amortization and recover {OOD} performance.