Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations Held in Vancouver, Canada and San Diego, California, USA on 14 December 2024 to 07 December 2025 Published as Volume 282 by the Proceedings of Machine Learning Research on 04 September 2026. Volume Edited by: Francisco Acosta Simone Azeglio Bahareh Tolooshams Chase van de Geijn Christian Shewmake Sophia Sanborn Nina Miolane Series Editors: Tegan Emerson Hoel Kervadec Neil D. Lawrence https://proceedings.mlr.press/v282/ Fri, 04 Sep 2026 16:46:03 +0000 Fri, 04 Sep 2026 16:46:03 +0000 Jekyll v3.10.0 In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors. However, it is not always clear *a priori* what geometric constraints are appropriate for a given task. Here, we explore the possibility that one can uncover useful geometric inductive biases by studying how training molds the Riemannian geometry induced by unconstrained neural network feature maps. We first show that at infinite width, neural networks with random parameters induce highly symmetric metrics on input space. This symmetry is broken by feature learning: networks trained to perform classification tasks learn to magnify local areas along decision boundaries. This holds in deep networks trained on high-dimensional image classification tasks, and even in self-supervised representation learning. These results begin to elucidate how training shapes the geometry induced by unconstrained neural network feature maps, laying the groundwork for an understanding of this richly nonlinear form of feature learning. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/zavatone-veth26a.html https://proceedings.mlr.press/v282/zavatone-veth26a.html Neural networks represent concepts, or “features”, but the general nature of these representations remains poorly understood. Previous approaches treat features as scalar-valued random variables. However, recent evidence for emergent world models motivates investigating when and how neural networks represent more complex structures. In this work, we formalize and study $\textit{feature fields}$—function-valued features defined over manifolds and other topological spaces corresponding to the underlying world (e.g., value functions, belief distributions). We introduce $\textit{linear field probing}$, a method that extends linear probing to extract feature fields from neural activations. Whereas a linear probe maps scalar features to individual points in activation space, a linear field probe embeds the topological space of a feature field into activation space. We prove that the geometry of this embedding fully defines the space of linearly representable functions for a given feature field. We empirically study feature fields of various topologies using linear field probing and present evidence of their emergence in transformers. This work establishes a formal connection between geometry and representation in neural networks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/yocum26a.html https://proceedings.mlr.press/v282/yocum26a.html Line graph transformation has been widely studied in graph theory, where each node in a line graph corresponds to an edge in the original graph. This has inspired a series of graph neural networks (GNNs) applied to transformed line graphs, which have proven effective in various graph representation learning tasks. However, there is limited theoretical study on how line graph transformation affects the expressivity of GNN models. In this study, we focus on two types of graphs known to be challenging to the Weisfeiler-Leman (WL) tests: Cai-Fürer-Immerman (CFI) graphs and strongly regular graphs, and show that applying line graph transformation helps exclude these challenging graph properties, thus potentially assist WL tests in distinguishing these graphs. We empirically validate our findings by conducting a series of experiments that compare the accuracy and efficiency of graph isomorphism tests and GNNs on both line-transformed and original graphs across these graph structure types. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/yang26a.html https://proceedings.mlr.press/v282/yang26a.html Data geometry is fundamental to machine learning and data analysis, yet practical tools for characterizing the geometry of data manifolds remain limited. While intrinsic dimension estimation is well-studied, curvature, a key measure of local manifold structure, is far harder to approximate from noisy, sparsely sampled data. We introduce a diffusion-based framework for curvature estimation aiming to mitigate challenges due to low sample density. We train a diffusion model to learn a latent representation of the manifold, which we then probe to augment the raw dataset and obtain a denser sample. Compared to state-of-the-art curvature estimators applied directly to the raw data, diffusion-augmented methods achieve superior performance on heterogeneous manifolds when using high-fidelity diffusion models. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/wang26e.html https://proceedings.mlr.press/v282/wang26e.html Algorithmic fairness aims to ensure the safe and responsible use of machine learning tools in applications across domains. In graph learning, several “fair rewiring” approaches have been proposed that perturb edges in the input graph to mitigate feature and relational bias. However, these approaches can lead to a decrease in accuracy in downstream tasks. On the other hand, classical rewiring approaches improve accuracy by mitigating over-smoothing and over-squashing effects induced by the graph’s topology. In this work we show that those classical rewiring approaches reinforce existing topological biases and boost accuracy at the cost of fairness. We propose a novel fairness metric (topological bias) that allows for evaluating relational bias separately from feature bias. We then propose a fairness constraint that can be incorporated into classical rewiring techniques to mitigate topological bias. We show that the resulting fairness-constrained rewiring balances fairness and accuracy effectively in graph learning tasks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/wang26d.html https://proceedings.mlr.press/v282/wang26d.html Understanding which parameter changes leave a Transformer’s function unchanged is essential for model comparison, optimization, and interpretability. This paper establishes a quantitative correspondence between geometric and algebraic approaches to neural network invariance, unifying two previously disconnected mathematical frameworks. We prove that Fisher-Rao curvature on the parameter-to-function quotient for multi-head attention provides a lower bound for bispectral energy in a linearized regime, revealing these two invariants as complementary aspects of the same underlying structure. Our theoretical framework yields practical benefits: a hybrid computational pipeline that substantially reduces runtime relative to pure algebraic methods while maintaining high discrimination accuracy for equivalence testing. Empirical validation across model scales from 4 to 24 heads demonstrates 98.9% validity of the theoretical bound, with the correspondence persisting through 10,000 training steps. By bridging differential geometry and harmonic analysis, we provide both theoretical insight into Transformer symmetries and efficient algorithms for identifying functionally equivalent models in practice. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/wang26c.html https://proceedings.mlr.press/v282/wang26c.html We give a geometry-first account of Transformers with GeLU. Building on a companion NeurReps paper that completely characterizes the head-wise gauge symmetries of multi-head attention, we treat the maximal head-wise symmetry group as given and study the induced geometry on the resulting quotient of functionally distinct models. On a generic regular set of parameters, this symmetry group acts freely and properly, so the parameter space fibers over a quotient manifold with gauge orbits as fibers. We establish an Ehresmann connection using the ambient Euclidean metric, which resolves the degeneracy of the Fisher–Rao (FR) metric along gauge directions. This framework clarifies that the natural gradient is the horizontal Riesz representative of the Euclidean gradient with respect to the FR geometry on the quotient. We show the connection has generically nonzero curvature, implying path-dependent holonomy in parameter updates. We also clarify the roles of the Attention (MHA) and FFN blocks: while MHA parameters possess gauge symmetry, FFN gradients are strictly horizontal as the FFN parameters are invariant under the MHA gauge group. We turn these ideas into practical diagnostics—a gauge-aware gradient split and a small-loop holonomy estimator—and report consistency checks aligning with the theory. Architectural choices such as RoPE appear as principled gauge reductions (e.g., per-head Q/K dimension from $d_k^2$ to $d_k$). Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/wang26b.html https://proceedings.mlr.press/v282/wang26b.html Modern Transformers possess redundant parameter symmetries that leave their function unchanged. We establish the complete gauge group structure for the canonical Transformer family, which encompasses standard architectures including GPT-2, BERT, LLaMA, and Qwen. For canonical Transformers with standard multi-head attention, we prove global maximality: the gauge group equals exactly $G_{max} = ((GL(d_k))^h \times (GL(d_v))^h) \rtimes S_h$ on the generic stratum where projection matrices have full column rank and head-wise attention controllability holds. For architectures with rotary position embeddings (RoPE) or relative encodings, as used in LLaMA and Qwen, the gauge group becomes $G_{RoPE} = ((C_{RoPE})^h \times (GL(d_v))^h) \rtimes S_h$ where $C_{RoPE}$ is the commutant of the position-dependent rotations—typically reducing to $(GL(1,\mathbb{C}))^{d_k/2}$ for standard RoPE implementations. We prove maximality through three key results: characterizing the Lie algebra of infinitesimal symmetries as $\mathfrak{g}_{max} = \bigoplus_{i=1}^h \mathfrak{gl}(d_k) \oplus \bigoplus_{i=1}^h \mathfrak{gl}(d_v)$ for canonical models, establishing that attention weights must be preserved up to head permutation under gauge equivalence, and demonstrating that query–key and value–output transformations necessarily factorize independently. These gauge symmetries persist through LayerNorm and extend to complete architectures, with the full model gauge group being $G_{Model} = \prod_{l=1}^L G_{Layer}^{(l)}$ Our characterization reveals over 1.1 million redundant dimensions in a 110M parameter Transformer Base model. Experiments on pretrained GPT-2 models from 124M to 1.5B parameters confirm that valid gauge transformations preserve model outputs to machine precision, while invalid transformations produce large errors, empirically supporting maximality. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/wang26a.html https://proceedings.mlr.press/v282/wang26a.html Noether’s theorem provides a powerful link between continuous symmetries and conserved quantities for systems governed by some variational principle. Perhaps unfortunately, most dynamical systems of interest in neuroscience and artificial intelligence cannot be described by any such principle. On the other hand, nonequilibrium physics provides a variational principle that describes how fairly generic noisy dynamical systems are most likely to transition between two states; in this work, we exploit this principle to apply Noether’s theorem, and hence learn about how the continuous symmetries of dynamical systems constrain their most likely trajectories. We identify analogues of the conservation of energy, momentum, and angular momentum, and briefly discuss examples of each in the context of models of decision-making, recurrent neural networks, and diffusion generative models. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/vastola26b.html https://proceedings.mlr.press/v282/vastola26b.html Dimensionality reduction algorithms like principal component analysis (PCA) are workhorses of machine learning and neuroscience, but each has well-known limitations. Variants of PCA are simple and interpretable, but not flexible enough to capture nonlinear data manifold structure. More flexible approaches have other problems: autoencoders are generally difficult to interpret, and graph-embedding-based methods can produce pathological distortions in manifold geometry. Motivated by these shortcomings, we propose a variational framework that casts dimensionality reduction algorithms as solutions to an optimal manifold embedding problem. By construction, this framework permits nonlinear embeddings, allowing its solutions to be more flexible than PCA. Moreover, the variational nature of the framework has useful consequences for interpretability: each solution satisfies a set of partial differential equations, and can be shown to reflect symmetries of the embedding objective. We discuss these features in detail and show that solutions can be analytically characterized in some cases. Interestingly, one special case exactly recovers PCA. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/vastola26a.html https://proceedings.mlr.press/v282/vastola26a.html A combinatorial neural code is a subset of the power set $2^{[n]}$ on $[n]=\{1,…, n\}$, in which each $1\leq i\leq n$ represents a neuron and each element (codeword) represents the co-firing event of some neurons. Consider a space $X\subseteq\mathbb{R}^d$, simulating an animal’s environment, and a collection $\mathcal{U}=\{U_1,\ldots ,U_n\}$ of open subsets of $X$. Each $U_i\subseteq X$ simulates a place field which is a specific region where a place cell $i$ is active. Then, the code of $\mathcal{U}$ in $X$ is defined as $\textnormal{code}(\mathcal{U},X)=\{ \sigma\subseteq[n]\bigg|\bigcap_{i\in\sigma} U_i\setminus\bigcup_{j\notin\sigma}U_j\neq\varnothing \}$. If a neural code $\mathcal{C}=\mbox{code}(\mathcal{U},X)$ for some $X$ and $\mathcal{U}$, we say $\mathcal{C}$ has a realization of open subsets of some space $X$. Although every combinatorial neural code obviously has a realization by some open subsets, determining whether it has a realization by some open convex subsets remains unsolved. Many studies attempted to tackle this decision problem, but only partial results were achieved. In fact, a previous study showed that the decision problem of convex neural codes is NP-hard. Furthermore, the authors of this study conjectured that every convex neural code can be realized as a minor of a neural code arising from a representable oriented matroid, which can lead to an equivalence between convex and polytope convex neural codes. Even though this conjecture has been confirmed in dimension two, its validity in higher dimensions is still unknown. To advance the investigation of this conjecture, we provide a complete characterization of the covering relations within the poset $\mathbf{P_{Code}}$ of neural codes. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/trang26a.html https://proceedings.mlr.press/v282/trang26a.html Representational similarity analysis (RSA) is widely used to analyze the alignment between humans and neural networks; however, conclusions based on this approach can be misleading without considering the underlying representational geometry. Our work introduces a framework using Ollivier Ricci Curvature and Ricci Flow to analyze the fine-grained local structure of representations. This approach is agnostic to the source of the representational space, enabling a direct geometric comparison between human behavioral judgments and a model’s vector embeddings. We apply it to compare human similarity judgments for 2D and 3D face stimuli with a baseline 2D-native network (VGG-Face) and a variant of it aligned to human behavior. Our results suggest that geometry-aware analysis provides a more sensitive characterization of discrepancies and geometric dissimilarities in the underlying representations that remain only partially captured by RSA. Notably, we reveal geometric inconsistencies in the alignment when moving from 2D to 3D viewing conditions. This highlights how incorporating geometric information can expose alignment differences missed by traditional metrics, offering deeper insight into representational organization. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/torbati26a.html https://proceedings.mlr.press/v282/torbati26a.html Batch size strongly influences optimisation, yet its role in non-Euclidean learning remains poorly understood. We propose \textbf{HGCNet}, a causally inspired hypergraph-based Deep Structural Causal Model that treats batch size as an intervention and organises its effects through stochastic mediators (gradient noise, sharpness, complexity) and a geometric proxy via Ollivier–Ricci curvature. Curvature is endogenous to the training recipe and, together with a curvature-aware regulariser, serves as a diagnostic of geometric stability rather than an isolated intervention. Experiments on graph and text benchmarks show consistent $2$–$4%$ accuracy improvements over strong baselines, providing the first causally structured analysis of how batch size shapes generalisation beyond vision. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/sun26b.html https://proceedings.mlr.press/v282/sun26b.html Adversarial attack reveals the vulnerability of deep learning models. It is assumed that high curvature may give rise to rough decision boundary and thus result in less robust models. However, the most commonly used \textit{curvature} is the curvature of loss function, scores or other parameters from within the model as opposed to decision boundary curvature, since the former can be relatively easily formed using second order derivative. In this paper, we propose a new query-efficient method, dynamic curvature estimation (DCE), to estimate the decision boundary curvature in a black-box setting. Our approach is based on CGBA, a black-box adversarial attack. By performing DCE on a wide range of classifiers, we discovered, statistically, a connection between decision boundary curvature and adversarial robustness. We also propose a new attack method, curvature dynamic black-box attack (CDBA) with improved performance using the estimated curvature. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/sun26a.html https://proceedings.mlr.press/v282/sun26a.html The study of neural representations, both in biological and artificial systems, is increasingly revealing the importance of geometric and topological structures. Inspired by this, we introduce Event2Vec, a novel framework for learning representations of discrete event sequences. Our model leverages a simple, additive recurrent structure to learn composable, interpretable embeddings. We provide a theoretical analysis demonstrating that, under specific training objectives, our model’s learned representations in a Euclidean space converge to an ideal additive structure. This ensures that the representation of a sequence is the vector sum of its constituent events, a property we term the linear additive hypothesis. To address the limitations of Euclidean geometry for hierarchical data, we also introduce a variant of our model in hyperbolic space, which is naturally suited to embedding tree-like structures with low distortion. We present experiments to validate our hypothesis and demonstrate the benefits of each geometry, highlighting the improved performance of the hyperbolic model on hierarchical event sequences. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/sulc26a.html https://proceedings.mlr.press/v282/sulc26a.html Oversquashing is a major hurdle to the application of geometric deep learning and graph neural networks to real world applications. Recent work has found connections between oversquashing and commute times, effective resistance, and the eigengap (or spectral gap) of the underlying graph. Graph rewiring is the most promising technique to alleviate this issue. Some prior work adds edges locally to highly negatively curved subgraphs. These local changes, however, have a small effect on global statistics such as commute times and the eigengap. Other prior work uses the spectrum of the graph Laplacian to target rewiring to increase the eigengap. These approaches, however, make large structural and topological changes to the underlying graph. We use ideas from geometric group theory to present \textsc{RelWire}, a rewiring technique based on the geometry of the graph. We explore topological properties of different rewiring techniques and show that \textsc{RelWire} is Pareto optimal: it has the best balance between improvement in eigengap and commute times and minimizing changes in the topology of the underlying graph, while performing comparably well on downstream tasks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/sonthalia26a.html https://proceedings.mlr.press/v282/sonthalia26a.html Predictive coding (PC) replaces global backpropagation with local optimization over weights and activations. We show that linear PC networks admit a natural formulation as cellular sheaves: the sheaf coboundary maps activations to edge-wise prediction errors, and PC inference is diffusion under the sheaf Laplacian. Sheaf cohomology then characterizes irreducible error patterns that inference cannot remove. We analyze recurrent topologies where feedback loops create internal contradictions, introducing prediction errors unrelated to supervision. Using a Hodge decomposition, we determine when these contradictions cause learning to stall. The sheaf formalism provides both diagnostic tools for identifying problematic network configurations and design principles for effective weight initialization for recurrent PC networks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/seely26a.html https://proceedings.mlr.press/v282/seely26a.html Neural ordinary differential equations (neural ODE) are powerful continuous-time machine learning models for depicting the behavior of complex dynamical systems, but their verification remains challenging due to limited reachability analysis tools adapted to them. We propose a novel interval-based reachability method that leverages continuous-time mixed monotonicity techniques for dynamical systems to compute an over-approximation for the neural ODE reachable sets. By exploiting the geometric structure of full initial sets and their boundaries via the homeomorphism property, our approach ensures efficient bound propagation. By embedding neural ODE dynamics into a mixed monotone system, our interval-based reachability approach, implemented in TIRA with single-step, incremental, and boundary-based approaches, provides sound and computationally efficient over-approximations compared with CORA’s zonotopes and NNV2.0 star set representations, while trading tightness for efficiency. This trade-off makes our method particularly suited for high-dimensional, real-time, and safety-critical applications. Applying mixed monotonicity to neural ODE reachability analysis paves the way for lightweight formal analysis by leveraging the symmetric structure of monotone embeddings and the geometric simplicity of interval boxes, opening new avenues for scalable verification. This novel approach is illustrated on two numerical examples of a spiral system and a fixed-point attractor system modeled as a neural ODE. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/sayed26a.html https://proceedings.mlr.press/v282/sayed26a.html In the algorithmic (Kolmogorov) view, agents are programs that track and compress sensory streams using generative programs. We propose a framework where the relevant structural prior is simplicity (Solomonoff) as \emph{compositional symmetry}, where natural streams are well described by (local) actions of finite-parameter Lie pseudogroups on geometrically and topologically complex low-dimensional configuration manifolds (latent spaces). Modeling the agent as a generic neural dynamical system coupled to such streams, we show that accurate world-tracking imposes (i) \emph{structural} constraints (equivariance of the agent system constitutive equations and readouts) and (ii) \emph{dynamical} constraints: under static inputs, symmetry induces conserved quantities (Noether-style labels) in agent dynamics and confines trajectories to reduced invariant manifolds; under slow drift, these manifolds move but remain low-dimensional. This yields a hierarchy of reduced manifolds aligned with the compositional factorization of the pseudogroup—a geometric account of the “blessing of compositionality” in deep models. We connect these ideas, at a high level, to the Spencer formalism for Lie pseudogroups, and formulate a symmetry-based, self-contained version of predictive coding in which higher layers receive only \emph{coarse-grained residual transformations} (prediction-error coordinates) along symmetry directions unresolved at lower layers. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/ruffini26a.html https://proceedings.mlr.press/v282/ruffini26a.html Associative memory architectures such as the Hopfield network have long been important conceptual and theoretical models for neuroscience and artificial intelligence. However, translating these abstract models into spiking neural networks has been surprisingly difficult. Indeed, much previous work has been restricted to storing a small number of primarily non-overlapping memories in large networks, thereby limiting their scalability. Here, we revisit the associative memory problem in light of recent advances in understanding spike-based computation. Using a recently-established geometric framework, we show that the spiking activity for a large class of all-inhibitory networks is situated on a low-dimensional, convex, and piecewise-linear manifold, with dynamics that move along the manifold. We then map the associative memory problem onto these dynamics, and demonstrate how the vertices of a hypercubic manifold can be used to store stable, overlapping activity patterns with a direct correspondence to the original Hopfield model. We propose several learning rules, and demonstrate a linear scaling of the storage capacity with the number of neurons, as well as robust pattern completion abilities. Overall, this work serves as a case study to demonstrate the effectiveness of using a geometrical perspective to design dynamics on neural manifolds, with implications for neuroscience and machine learning. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/podlaski26a.html https://proceedings.mlr.press/v282/podlaski26a.html The goal of this paper is to address the problem of \textit{global} point cloud registration (PCR) i.e., finding the optimal alignment between point clouds irrespective of the initial poses of the scans. This problem is notoriously challenging for classical optimization methods due to computational constraints. First, we show that many state-of-the-art deep learning methods suffer from huge performance degradation when the point clouds are arbitrarily placed in space. We propose that \textit{equivariant deep learning} should be utilized for solving this task and we characterize the specific type of bi-equivariance of PCR. Then, we design BiEquiformer a novel and scalable \textit{bi-equivariant} pipeline i.e. equivariant to the independent transformations of the input point clouds. While a naive approach would process the point clouds independently we design expressive bi-equivariant layers that fuse the information from both point clouds. This allows us to extract high-quality superpoint correspondences and in turn, robust point-cloud registration. Extensive comparisons against state-of-the-art methods show that our method achieves comparable performance in the canonical setting and superior performance in the robust setting in both the 3DMatch and the challenging low-overlap 3DLoMatch dataset. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/pertigkiozoglou26a.html https://proceedings.mlr.press/v282/pertigkiozoglou26a.html The adoption of increasingly complex deep models has fueled an urgent need for insight into how these models make predictions. Counterfactual explanations form a powerful tool for providing actionable explanations to practitioners. Previously, counterfactual explanation methods have been designed by traversing the latent space of generative models. Yet, these latent spaces are usually greatly simplified, with most of the data distribution complexity contained in the decoder rather than the latent embedding. Thus, traversing the latent space naively without taking the nonlinear decoder into account can lead to unnatural counterfactual trajectories. We introduce counterfactual explanations obtained using a Riemannian metric pulled back via the decoder and the classifier under scrutiny. This metric encodes information about the complex geometric structure of the data and the learned representation, enabling us to obtain robust counterfactual trajectories with high fidelity, as demonstrated by our experiments in real-world tabular datasets. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/pegios26a.html https://proceedings.mlr.press/v282/pegios26a.html Low-Rank Adaptation (LoRA) has been widely adopted as a Parameter Efficient Fine-Tuning method for large models such as Large Language Models (LLMs) and Vision Transformer (ViT). However, it encounters scalability limitations, particularly in storage and deployment efficiency, when applied to large foundational models or a wide range of task-specific adaptations, due to the overhead of managing multiple adapters and the reliance on linearly constrained spaces for representation. To address these limitations, Fourier Fine-Tuning (FourierFT) has emerged as an alternative, leveraging the Fourier transform to achieve comparable or superior performance to LoRA while utilizing significantly fewer trainable parameters. Nevertheless, FourierFT targets the entire frequency spectrum to apply updates, which may cause inefficiency, particularly when the meaningful information is concentrated within a specific set of frequency components. The magnitude of each Fourier component reflects its contribution to the original weight update. Thus, selecting Top-K components with the highest magnitudes effectively captures the most informative changes. Therefore, we propose Layerwise Fourier Masked Adapter (LFMA), which selectively fine-tunes using Top-K informative frequency components and resulting in an enhancement of both parameter efficiency and task-specific adaptation. Empirically, we showed similar or better performance than FourierFT in four tasks: image classification, instruction tuning, natural language generation, and natural language understanding. These results demonstrate that selectively fine-tuning in the most informative frequency components is able to push the limits of adapter-based fine-tuning further in terms of scalability and expressivity. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/park26a.html https://proceedings.mlr.press/v282/park26a.html Graph Neural Networks (GNN) can capture the geometric properties of neural representations in EEG data. Here we utilise those to study how reinforcement-based motor learning affects neural activity patterns during motor planning, leveraging the inherent graph structure of EEG channels to capture the spatial relationships in brain activity. By exploiting task-specific symmetries, we define different pretraining strategies that not only improve model performance across all participant groups but also validate the robustness of the geometric representations. Explainability analysis based on the graph structures reveals consistent group-specific neural signatures that persist across pretraining conditions, suggesting stable geometric structures in the neural representations associated with motor learning and feedback processing. These geometric patterns exhibit partial invariance to certain task space transformations, indicating symmetries that enable generalisation across conditions while maintaining specificity to individual learning strategies. This work demonstrates how GNNs can uncover the effects of previous outcomes on motor planning, in a complex real-world task, providing insights into the geometric principles governing neural representations. Our experimental design bridges the gap between controlled experiments and ecologically valid scenarios, offering new insights into the organisation of neural representations during naturalistic motor learning, which may open avenues for exploring fundamental principles governing brain activity in complex tasks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/nardi26a.html https://proceedings.mlr.press/v282/nardi26a.html A common practice when training Deep Neural Networks is to force the learned representations to lie on the standard unit hypersphere, with respect to the $L_2$ norms. Such practice has been shown to improve both the stability and final performances of DNNs in many applications. In this paper, we derive a unified theoretical framework for learning representation on any $L_p$ hyperspheres for classification tasks, based on Maximum A Posteriori (MAP) modeling. Specifically, we give an expression of the probability distribution of multivariate Gaussians projected on any $L_p$ hypersphere and derive the general associated loss function. Additionally, we show that this framework demonstrates the theoretical equivalence of all projections on $L_p$ hyperspheres through the MAP modeling. It also provides a new interpretation of traditional Softmax Cross Entropy with temperature (SCE-$\tau$) loss functions. Experiments on standard computer vision datasets give an empirical validation of the equivalence of projections on $L_p$ unit hyperspheres when using adequate objectives. It also shows that the SCE-$\tau$ on projected representations, with optimally chosen temperature, shows comparable performances. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/michel26a.html https://proceedings.mlr.press/v282/michel26a.html This paper explores a novel brain decoding model for 3D shape perception through a dual pathway architecture mirroring biological vision. Our bio-inspired approach decomposes 3D visual processing into object identity (ventral pathway) and spatial orientation (dorsal pathway) during continuous rotations. We employ circular regression for angle prediction and develop EEG-conditioned multiview diffusion for 3D reconstruction. Our approach successfully decodes both object identity and spatial orientation from EEG signals and demonstrates feasible 3D reconstruction from neural activity, with interpretability analysis revealing ventral pathway activation patterns that support object recognition performance. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/masclef26a.html https://proceedings.mlr.press/v282/masclef26a.html We develop K-theoretic persistent cohomology (KPCH): a principled extension of 1-parameter persistent (co)homology that equips the Grothendieck group of persistence modules with lambda-operations arising from exterior powers. This yields new, computable persistence layers that quantify concurrency among cohomology classes via interval intersections. We establish the core algebraic and stability results, provide an interval-calculus for efficient computation on barcodes, and demonstrate empirical benefits on graph filtrations, where KPCH separates patterns that standard additive H^p summaries such as total persistence cannot distinguish. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/maruyama26b.html https://proceedings.mlr.press/v282/maruyama26b.html We propose Homological Representation Learning (HomRL), an architecture-agnostic regularization method for graph encoders that aligns latent embeddings with an efficiently computable homological signature of the input. In this paper, we give both theoretical results on representation invariance bounds and empirical results on molecular graph classification tasks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/maruyama26a.html https://proceedings.mlr.press/v282/maruyama26a.html Hyperbolic neural networks (HNNs) have been proved effective in modeling complex data structures. However, previous works mainly focused on the Poincaré ball model and the hyperboloid model as coordinate representations of the hyperbolic space, often neglecting the Klein model. Despite this, the Klein model offers its distinct advantages thanks to its straight-line geodesics, which facilitates the well-known Einstein midpoint construction, previously leveraged to accompany HNNs in other models. In this work, we introduce a framework for hyperbolic neural networks based on the Klein model. We provide a detailed formulation for representing useful operations using the Klein model. We further study the Klein linear layer and prove that the “tangent space construction” of the scalar multiplication and parallel transport are exactly the Einstein scalar multiplication and the Einstein addition, analogous to the Möbius operations used in the Poincaré ball model. We show numerically that Klein HNNs perform on par with HNNs using the Poincaré ball model, providing a third option for HNN that works as a building block for more complicated architectures. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/mao26a.html https://proceedings.mlr.press/v282/mao26a.html Transformer models exhibit remarkable in-context learning (ICL), adapting to novel tasks from examples within their context, yet the underlying mechanisms remain largely mysterious. Here, we provide an exact analytical characterization of ICL emergence by deriving the closed-form stochastic gradient descent (SGD) dynamics for a simplified linear transformer performing regression tasks. Our analysis reveals key properties: (1) a natural separation of timescales directly governed by the input data’s covariance structure, leading to staged learning; (2) an exact description of how ICL develops, including fixed points corresponding to learned algorithms and conservation laws constraining the dynamics; and (3) surprisingly nonlinear learning behavior despite the model’s linearity. We hypothesize this phenomenology extends to non-linear models. To test this, we introduce theory-inspired macroscopic measures (spectral rank dynamics, subspace stability) and use them to provide mechanistic explanations for (1) the sudden emergence of ICL in attention-only networks and (2) delayed generalization (grokking) in modular arithmetic models. Our work offers an exact dynamical model for ICL and theoretically grounded tools for analyzing complex transformer training. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/mainali26a.html https://proceedings.mlr.press/v282/mainali26a.html Most equivariant neural networks rely on a single global symmetry, limiting their use in domains where symmetries are instead local. We introduce Torsor CNNs, a framework for learning on graphs with local symmetries encoded as edge potentials—group-valued transformations between neighboring coordinate frames. We establish that this geometric construction is fundamentally equivalent to the classical group synchronization problem, yielding: (1) a Torsor Convolutional Layer that is provably equivariant to local changes in coordinate frames, and (2) the frustration loss—a standalone geometric regularizer that encourages locally equivariant representations when added to any NN’s training objective. The Torsor CNN framework unifies and generalizes several architectures—including classical CNNs and Gauge CNNs on manifolds—by operating on arbitrary graphs without requiring a global coordinate system or smooth manifold structure. We establish the mathematical foundations of this framework and demonstrate its applicability to multi-view 3D recognition, where relative camera poses naturally define the required edge potentials. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/li26a.html https://proceedings.mlr.press/v282/li26a.html Many natural phenomena are characterized by self-similarity, for example the symmetry of human faces, or a repetitive motif of a song. Studying of such symmetries will allow us to gain deeper insights into the underlying mechanisms of complex systems. Recognizing the importance of understanding these patterns, we propose a geometrically inspired framework to study such phenomena in artificial neural networks. To this end, we introduce \emph{CantorNet}, inspired by the triadic construction of the Cantor set, which was introduced by Georg Cantor in the $19^\text{th}$ century. In mathematics, the Cantor set is a set of points lying on a single line that is self-similar and has a counter intuitive property of being an uncountably infinite null set. Similarly, we introduce CantorNet as a sandbox for studying self-similarity by means of novel topological and geometrical complexity measures. CantorNet constitutes a family of ReLU neural networks that spans the whole spectrum of possible Kolmogorov complexities, including the two opposite descriptions (linear and exponential as measured by the description length). CantorNet’s decision boundaries can be arbitrarily ragged, yet are analytically known. Besides serving as a testing ground for complexity measures, our work may serve to illustrate potential pitfalls in geometry-ignorant data augmentation techniques and adversarial attacks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/lewandowski26a.html https://proceedings.mlr.press/v282/lewandowski26a.html Recurrent neural networks (RNNs) are central to modeling neural computation in systems neuroscience, yet the principles that enable their stable and efficient training at large scales remain poorly understood. Seminal work in machine learning predicts that the effective learning rate should shrink with the size of feedforward networks. Here, we demonstrate an analogous phenomenon, termed learning rate collapse, in which the maximum trainable learning rate decreases inversely with the number of neurons. This behavior can be mitigated partially by scaling parameters with the inverse of network, though learning still takes longer for larger networks. These limits are further compounded by severe memory demands, which together make training large RNNs both unstable and computationally costly. As a proof of principle for mitigating learning rate collapse, we study the learning process of low-rank networks, which enforces a low-dimensional geometry in RNN representations. These results situate learning rate collapse within a broader lineage of scaling analyses in RNNs, with potential solutions likely to come from future work that incorporates careful consideration of symmetry and geometry in neural representations. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/kurtkaya26a.html https://proceedings.mlr.press/v282/kurtkaya26a.html The extended Gaussian family is the closure of the Gaussian family obtained by completing the Gaussian family with the counterpart elements induced by degenerate covariance or degenerate precision matrices, or a mix of both degeneracies. The parameter space of the extended Gaussian family forms a symmetric positive semi-definite matrix bicone, i.e. intersection of two partial symmetric positive semi-definite matrix cones. In this paper, we study the Hilbert geometry of such an open bounded convex symmetric positive-definite bicone. We report the closed-form formula for the corresponding Hilbert metric distance and study exhaustively its invariance properties. We also touch upon potential applications of this geometry for dealing with extended Gaussian distributions. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/karwowski26a.html https://proceedings.mlr.press/v282/karwowski26a.html Visual representations in the human brain are shaped by the pressure to support planning and interactions with the environment. Do visual representations in deep network models converge with visual representations in humans? Here, we investigate this question for a new class of effective self-supervised models: Masked Autoencoders (MAEs). We compare image MAEs and video MAEs to neural responses in humans as well as convolutional neural networks. The results reveal that representations learned by MAEs diverge from neural representations in humans and convolutional neural networks. Fine-tuning MAEs with a supervised task improves their correspondence with neural responses but is not sufficient to bridge the gap that separates them from supervised convolutional networks. Finally, video MAEs show closer correspondence to neural representations than image MAEs, revealing an important role of temporal information. However, convolutional networks based on optic flow show a closer correspondence to neural responses in humans than even video MAEs, indicating that while masked autoencoding yields visual representations that are effective at multiple downstream tasks, it is not sufficient to learn representations that converge with human vision. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/karimi26a.html https://proceedings.mlr.press/v282/karimi26a.html Convolutional Neural Networks exhibit inherent equivariance to image translation, leading to efficient parameter and data usage, faster learning, and improved robustness. The concept of translation equivariant networks has been successfully extended to rotation transformation using group convolution for discrete rotation groups and harmonic functions for the continuous rotation group encompassing $360^\circ$. We explore the compatibility of the Self-Attention mechanism with full rotation equivariance, in contrast to previous studies that focused on discrete rotation. We introduce the Harmformer, a harmonic transformer with a convolutional stem that achieves equivariance for both translation and continuous rotation. Accompanied by an end-to-end equivariance proof, the Harmformer not only outperforms previous equivariant transformers, but also demonstrates inherent stability under any continuous rotation, even without seeing rotated samples during training. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/karella26a.html https://proceedings.mlr.press/v282/karella26a.html Despite their success, modern convolutional neural networks (CNNs) exhibit fundamental limitations, including data inefficiency, poor out-of-distribution generalization, and vulnerability to adversarial perturbations. These shortcomings can be traced to a lack of inductive biases that reflect the inherent geometric structure of the visual world. The primate visual system, in contrast, demonstrates superior efficiency and robustness, suggesting that its architectural and computational principles,which evolved to internalize these structures,may offer a blueprint for more capable artificial vision. This paper introduces Visual Cortex Network (VCNet), a novel neural network architecture whose design is informed by the macro-scale organization of the primate visual cortex. VCNet is framed as a geometric framework that emulates key biological mechanisms, including hierarchical processing across distinct cortical areas, dual-stream information segregation for learning disentangled representations, and top-down predictive feedback for representation refinement. We interpret these mechanisms through the lens of geometry and dynamical systems, positing that they guide the learning of structured, low-dimensional neural manifolds. We evaluate VCNet on two specialized benchmarks: the Spots-10 animal pattern dataset, which probes sensitivity to natural textures, and a light field image classification task, which requires processing higher-dimensional visual data. Our results show that VCNet achieves state-of-the-art accuracy of 92.1% on Spots-10 and 74.4% on the light field dataset, surpassing contemporary models of comparable size. This work demonstrates that integrating high-level neuroscientific principles, viewed through a geometric lens, can lead to more efficient and robust models, providing a promising direction for addressing long-standing challenges in machine learning. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/hill26a.html https://proceedings.mlr.press/v282/hill26a.html Biological and artificial intelligence systems navigate the fundamental efficiency-robustness tradeoff for optimal encoding, i.e., they must efficiently encode numerous attributes of the input space while also being robust to noise. This challenge is particularly evident in hierarchical processing systems like the human brain. With a view towards understanding how systems navigate the efficiency-robustness tradeoff, we turned to a population geometry framework for analyzing representations in the human visual cortex alongside artificial neural networks (ANNs). In the ventral visual stream, we found general-purpose, scale-free representations characterized by a power law-decaying eigenspectrum in most areas. However, in certain higher-order visual areas did not have scale-free representations, indicating that scale-free geometry is not a universal property of the brain. In parallel, ANNs trained with a self-supervised learning objective also exhibited scale-free geometry, but not after fine-tuning on a specific task. Based on these empirical results and our analytical insights, we posit that a system’s representation geometry is not a universal property and instead depends upon the computational objective. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/ghosh26a.html https://proceedings.mlr.press/v282/ghosh26a.html In self-supervised contrastive learning, multiple augmentations of the same input naturally form a set of latent representations, or an orbit. Ideally, these representations should remain compact and directionally consistent under transformations. Standard methods such as SimCLR prioritize separating different samples but do not explicitly enforce intra-orbit coherence, allowing augmented views of the same input to drift in latent space. We propose Orbit Regularization Loss (ORL), a lightweight extension to the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss that reweights negative pairs based on a tension score - a measure of alignment between the positive-pair direction and the candidate negative’s displacement. This encourages augmented views to align along stable latent directions, reducing orbit spread without architectural changes or additional supervision. For now, ORL is aimed at improving the geometric structure of embeddings, rather than directly targeting downstream classification accuracy. Experiments on MNIST and CIFAR-10 show that ORL lowers intra-orbit variance, improves directional consistency, and yields a more coherent latent space geometry compared to the NT-Xent baseline. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/ghosal26a.html https://proceedings.mlr.press/v282/ghosal26a.html In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. For neural network models, visualizing how this loss changes as model parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) as well as global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher-dimensional loss landscapes. After describing this new topological landscape profile representation, we show how the shape of loss landscapes can reveal new details about model performance and learning dynamics, highlighting several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks). Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces: we find that the topology of the loss landscape is simpler for better-performing models; and we observe greater variation in the shape of loss landscapes near transitions from low to high model performance. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/geniesse26a.html https://proceedings.mlr.press/v282/geniesse26a.html We explore the topology of representation manifolds arising in autoregressive neural language models trained on raw text data. In order to study their properties, we introduce tools from computational algebraic topology, which we use as a basis for a measure of topological complexity, that we call \emph{perforation}. Using this measure, we study the evolution of topological structure in GPT based large language models across depth and time during training. We then compare these to gated recurrent models, and show that the latter exhibit more topological complexity, with a distinct pattern of changes common to all natural languages but absent from synthetically generated data. The paper presents a detailed analysis of the representation manifolds derived by these models based on studying the shapes of vector clouds induced by them as they are conditioned on sentences from corpora of natural language text. The methods developed in this paper are novel in the field and based on mathematical apparatus that might be unfamiliar to the target audience. To help with that we introduce the minimum necessary theory, and provide additional visualizations in the appendices. The main contribution of the paper is a striking observation about the topological structure of the transformer as compared to LSTM based neural architectures. It suggests that further research into mathematical properties of these neural networks is necessary to understand the operation of large transformer language models. We hope this work inspires further explorations in this direction within the NLP community. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/fitz26a.html https://proceedings.mlr.press/v282/fitz26a.html Dimensionality reduction is a key ingredient of many machine learning algorithms and is paramount to their success. For manifold-valued data, the nonlinear equivalent of the well-known principal component analysis (PCA), called, principal geodesic analysis (PGA) is used quite often. An alternative to PGA that is more general and flexible, called "Nested Homogeneous Spaces (NHS)" for dimensionality reduction of manifold-valued data was recently introduced. In this paper, we present a novel probabilistic version of the NHS model (PNHS) for dimensionality reduction of high dimensional manifold-valued data in Riemannian homogeneous spaces. The PNHS model has several advantages over its deterministic counterpart namely, the NHS model. In particular, the ability to, quantify uncertainty in parameter estimates and tackle missing data. We demonstrate these advantages via real and synthetic data examples. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/fan26a.html https://proceedings.mlr.press/v282/fan26a.html Attention-Deficit/Hyperactivity Disorder (ADHD) is a prevalent neurodevelopmental disorder widely hypothesized to stem from alterations in large-scale brain connectivity. However, neuroimaging studies have yielded inconsistent findings, motivating the need for advanced analytical methods capable of capturing the complex, dynamic nature of brain function. In this study, we apply Topological Data Analysis (TDA), specifically the Mapper algorithm, to resting-state functional magnetic resonance imaging (fMRI) data from the multi-site ADHD-200 dataset. We constructed graphical representations of brain state dynamics for participants with ADHD and typically developing controls (TDC) from three independent sites. The topological structure of these graphs was quantified using network centrality measures (betweenness, closeness, and degree). Our results reveal a significant increase in centrality measures in the ADHD group compared to TDC in three cohorts. Furthermore, we observed a weak but significant positive correlation between centrality and symptom severity in one of the cohorts. We conclude that TDA-derived centrality measures can detect alterations in the dynamical organization of brain activity in ADHD, potentially reflecting a less efficient or more rigid network topology. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/duman26a.html https://proceedings.mlr.press/v282/duman26a.html Knowledge distillation (KD) methods transfer the knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student’s backbone using feature-based losses \emph{exclusively} (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a \emph{knowledge quality metric} for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to $15$% over standard approaches. We publicly share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.git . Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/cooper26a.html https://proceedings.mlr.press/v282/cooper26a.html A key challenge in artificial intelligence and neuroscience is understanding how neural systems learn representations that capture the underlying dynamics of the world. Most world models represent the transition function with unstructured neural networks, limiting interpretability, sample efficiency, and generalization to unseen states or action compositions. We address these issues with a generalizable world model grounded in \textit{Vector Symbolic Architecture} (VSA) principles as geometric priors. Our approach utilizes learnable Fourier Holographic Reduced Representation (FHRR) encoders to map states and actions into a high-dimensional complex vector space with learned group structure and models transitions with element-wise complex multiplication. We formalize the framework’s group-theoretic foundation and show how training such structured representations to be approximately invariant enables strong multi-step composition directly in latent space and generalization performances over various experiments. On a discrete grid world environment, our model achieves 87.5% zero-shot accuracy to unseen state-action pairs, obtains 53.6% higher accuracy on 20-timestep horizon rollouts, and demonstrates $4\times$ higher robustness to noise relative to an MLP baseline. These results highlight how training to have latent group structure yields generalizable, data-efficient, and interpretable world models, providing a principled pathway toward structured models for real-world planning and reasoning. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/chung26a.html https://proceedings.mlr.press/v282/chung26a.html Brain-computer interfaces (BCIs) offer transformative potential, but decoding neural signals presents significant challenges. The core premise of this paper is built around demonstrating methods to elucidate the underlying low-dimensional geometric structure present in high-dimensional brainwave data in order to assist in downstream BCI-related neural classification tasks. We demonstrate two pipelines related to electroencephalography (EEG) signal processing: (1) a preliminary pipeline removing noise from individual EEG channels, and (2) a downstream manifold learning pipeline uncovering geometric structure across networks of EEG channels. We conduct preliminary validation using two EEG datasets and situate our demonstration in the context of the BCI-relevant imagined digit decoding problem. Our preliminary pipeline uses an attention-based EEG filtration network to extract clean signal from individual EEG channels. Our primary pipeline uses a fast Fourier transform, a Laplacian eigenmap, a discrete analog of Ricci flow via Ollivier’s notion of Ricci curvature, and a graph convolutional network to perform dimensionality reduction on high-dimensional multi-channel EEG data in order to enable regularizable downstream classification. Our system achieves competitive performance with existing signal processing and classification benchmarks; we demonstrate a mean test correlation coefficient of $>$0.95 at 2 dB on semi-synthetic neural denoising and a downstream EEG-based classification accuracy of 0.97 on distinguishing digit- versus non-digit thoughts. Results are preliminary and our geometric machine learning pipeline should be validated by more extensive follow-up studies; generalizing these results to larger inter-subject sample sizes, different hardware systems, and broader use cases will be crucial. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/choi26a.html https://proceedings.mlr.press/v282/choi26a.html We establish a mathematical connection between neural field optimization and Transformer attention mechanics. First, we prove that Transformer-based operators learning a neural field are equivariant to affine transformations (translations and positive scalings) when using relative positional encodings and coordinate normalization, extending geometric deep learning to meta-learning of continuous functions. Second, we demonstrate that linear attention is an exact computation of the negative gradient of squared-error loss for sinusoidal neural fields, with softmax attention shown empirically and theoretically to converge to such an identity at rate $O(\tau^{-2})$ as temperature scales. The novel results reveal that attention mechanisms have an implicit geometric encoding that is well-suited to learn continuous functions. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/cherukuri26a.html https://proceedings.mlr.press/v282/cherukuri26a.html Extraction of structure, in particular of group symmetries, is increasingly crucial to understanding and building intelligent models. In particular, some information-theoretic models of parsimonious learning have been argued to induce invariance extraction. Here, we formalise these arguments from a group-theoretic perspective. We then extend them to the study of more general probabilistic symmetries, through compressions preserving geometric measures of complexity. More precisely, our framework implements a trade-off between compression and preservation of the divergence from a given hierarchical model, yielding a novel generalisation of the Information Bottleneck framework. Through appropriate choices of hierarchical models, we fully characterise (in the discrete and full support case) channel invariance, channel equivariance and distribution invariance under permutation. Allowing imperfect divergence preservation then leads to principled definitions of “soft symmetries”, where the “coarseness” corresponds to the degree of compression of the system. In simple synthetic experiments, we demonstrate that our method successively recovers, at increasingly compressed “resolutions”, nested but increasingly perturbed equivariances, where new equivariances emerge at bifurcation points of the trade-off parameter. Our framework suggests a new path for the extraction of generalised probabilistic symmetries. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/charvin26a.html https://proceedings.mlr.press/v282/charvin26a.html Memorization in generative text-to-image diffusion models is a phenomenon where instead of valid image generations, the model outputs near-verbatim reproductions of training images. This poses privacy and copyright risks, and remains difficult to prevent without harming prompt fidelity. We present a mid-generation, geometry-informed criterion that detects, and then helps avoid (mitigate), memorized outputs. Our method analyzes the natural image distribution manifold as learnt by the diffusion model. We analyze a memorization criterion that has a local curvature interpretation. Thus we can track the generative process, and our criterion’s trajectory throughout it, to understand typical geometrical structures traversed throughout this process. This is harnessed as a geometry-aware indicator that distinguishes memorized from valid generations. Notably, our criterion uses only the direction of the normalized score field, unlike prior magnitude-based methods; combining direction and magnitude we improve mid-generation detection SOTA by $\sim 5%$. Beyond detecting memorization, we use this indicator as a plug-in to a mitigation policy to steer trajectories away from memorized basins while preserving alignment to the text. Empirically, this demonstrates improved fidelity–memorization trade-off over the competitors. By linking memorization to magnitude-invariant geometric signatures of the generative process, our work opens a new direction for understanding—and systematically mitigating—failure modes in diffusion models. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/brokman26a.html https://proceedings.mlr.press/v282/brokman26a.html Many tasks require mapping continuous input data (e.g. images) to discrete task outputs (e.g. class labels). Yet, how neural networks learn to perform such discrete computations on continuous data manifolds remains poorly understood. Here, we show that signatures of such computations emerge in the representational geometry of neural networks as they learn. By analysing the Riemannian pullback metric across layers of a neural network, we find that network computation can be decomposed into two functions: discretising continuous input features and performing logical operations on these discretised variables. Furthermore, we demonstrate how different learning regimes (rich vs. lazy) have contrasting metric and curvature structures, affecting the ability of the networks to generalise to unseen inputs. Overall, our work provides a geometric framework for understanding how neural networks learn to perform discrete computations on continuous manifolds. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/brandon26a.html https://proceedings.mlr.press/v282/brandon26a.html Multi-head attention powers Transformer networks, the primary deep learning architecture behind the success of large language models (LLMs). Yet, the theoretical advantages of multi-head versus single-head attention, beyond mere parallel processing, remain underexplored. In this paper, we reframe multi-head attention as a system of potentially synergistic computational graphs, where each head functions as a feedforward directed acyclic graph (DAG) with a common sink state. We provide intuition and preliminary theoretical analysis of mixing time and minimax fidelity in this framework. Our results show that multi-head attention can synergistically enhance information propagation, yielding faster mixing times and minimax fidelity amplification under specific head-diversity conditions. Finally, we train single-head and multi-head Transformers, each with the same total number of parameters, on sequence manipulation tasks and empirically verify the predicted effects. The code is available at https://github.com/haitzsaezdeocariz/beyondparallelism. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/borde26a.html https://proceedings.mlr.press/v282/borde26a.html The brain’s ability to store and manipulate working memory (WM) sequences is pivotal for higher cognitive reasoning. Although the neural circuit mechanisms for storing WM sequences have been extensively studied, those for manipulating WM sequences remain largely unknown. Inspired by a recent WM sequence manipulation experiment in monkeys, we design a functional, biologically plausible neural circuit model that realizes WM sequence permutations using guidance from permutation groups and their Cayley graph representations. The circuit consists of two interconnected modules: a memory module composed of continuous attractor-based memory motifs that store and interchange items in WM sequences, and a control module that sends gain modulations to guide permutation operations within the memory module. The control module features a hierarchical tree structure that decomposes complex permutations into a sequence of basic two-item swaps, simplifying circuit implementations. We demonstrate that permutation circuit architectures have one-to-one correspondence with Cayley graphs representing permutation group structure, where the group generating set directly determines the connectivity between memory motifs. Since each permutation group may have multiple generating sets, there are multiple circuit architectures implementing the same permutation. We therefore utilize Cayley graph analysis to determine trade-offs between computational efficiency, circuit complexity, and circuit robustness. Our study establishes connections between abstract group theory, Cayley graphs, and biologically plausible circuit architectures, providing insights into principled circuit design via algebraic frameworks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/bien26a.html https://proceedings.mlr.press/v282/bien26a.html Efficiently processing structured point cloud data while preserving multiscale information is a key challenge across domains, from graphics to atomistic modeling. Using a curated dataset of simulated galaxy positions and properties, represented as point clouds, we benchmark the ability of graph neural networks to simultaneously capture local clustering environments and long-range correlations. Given the homogeneous and isotropic nature of the Universe, the data exhibits a high degree of symmetry. We therefore focus on evaluating the performance of Euclidean symmetry-preserving ($E(3)$-equivariant) graph neural networks, showing that they can outperform non-equivariant counterparts and domain-specific information extraction techniques in downstream performance as well as simulation-efficiency. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/balla26a.html https://proceedings.mlr.press/v282/balla26a.html Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/azeglio26a.html https://proceedings.mlr.press/v282/azeglio26a.html This paper explores a simple question: can we model the internal transformations of a neural network using dynamical systems theory? We introduce Koopman autoencoders to capture how neural representations evolve through network layers, treating these representations as states in a dynamical system. Our approach learns a surrogate model that predicts how neural representations transform from the input to the output, with two key advantages. First, by way of lifting the original states via an autoencoder, it operates in a linear space, making editing the dynamics straightforward. Second, it preserves the topology of the original representations by regularizing the autoencoding objective. We demonstrate that these surrogate models naturally replicate the progressive topological simplification observed in neural networks. As a practical application, we show how our approach enables targeted class unlearning in the Yin-Yang and MNIST classification tasks. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/aswani26a.html https://proceedings.mlr.press/v282/aswani26a.html Compositional generalization—the ability to understand novel combinations of familiar components—remains a significant challenge for neural networks despite their success in many language tasks. Current evaluation methods focus on behavioral measures that reveal \emph{when} models fail to generalize compositionally, but provide limited insight into \emph{why} these failures occur at the representational level. We introduce \textit{Homomorphism Error} (HE), a structural metric that quantifies how well neural network representations preserve compositional operations by measuring deviations from approximate homomorphisms between expression spaces and their internal representations. Through controlled experiments on SCAN-style synthetic compositional tasks and small-scale Transformers, we demonstrate that HE serves as a strong predictor of out-of-distribution generalization performance, achieving $R^2 = 0.73$ correlation with OOD compositional generalization accuracy. Furthermore, our analysis reveals that model architecture has minimal impact on compositional structure, training data coverage exhibits threshold effects, but noise injection systematically degrades compositional representations in predictable ways. Importantly, we find that different aspects of compositionality—unary operations (modifiers) versus binary operations (sequence composition)—exhibit distinct sensitivities to distributional shifts, with modifier representations being particularly vulnerable to spurious correlations. These findings provide new mechanistic insights into compositional learning and establish homomorphism error as a valuable diagnostic tool for developing more robust neural architectures training methods. Code and data will be made publicaly available. Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/an26a.html https://proceedings.mlr.press/v282/an26a.html Fri, 04 Sep 2026 00:00:00 +0000 https://proceedings.mlr.press/v282/acosta26a.html https://proceedings.mlr.press/v282/acosta26a.html