MetaOthello: A Controlled Study of Multiple World Models in Transformers

Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting "world models". Previous experiments on Othello-playing neural networks test world-model learning, but focus on a single game with a single set of rules. We introduce MetaOthello, a controlled suite of Othello-like games with shared syntax but different rules or tokenizations, and train small GPTs on mixed-variant data. We show that transformers trained on multiple Othello variants learn shared world-state representations: linear probes trained on one game intervene on another’s board state nearly as well as matched probes. When the games conflict, the model resolves the resulting ambiguity through a localized mechanism we identify and steer. For isomorphic games with token remapping, representations are equivalent up to a single orthogonal rotation that generalizes across layers, showing the shared structure is abstract rather than tied to surface form. Together, these results show that transformers reconcile conflicting world models by sharing structure and localizing conflict. MetaOthello thus offers a path toward understanding how transformers organize many world models at once.