Roots for Verbs: Causal Analysis Reveals Verb Centered Syntax Where Geometric Analysis Does Not

Abstract

Most mechanistic interpretability methods rely on the Linear Representation Hypothesis: the assumption that linguistic features are encoded in approximately linear subspaces. We challenge this assumption by introducing a non-linear variant of Distributed Alignment Search (DAS) based on geodesic interpolation in hyperbolic space. Unlike standard DAS, our method performs partial, geometry-aware interventions rather than hard coordinate swaps. Using controlled syntactic test suites from SyntaxGym, which allow downstream effects on post-verbal tokens to be measured, we find that non-linear DAS reliably recovers strong verb-dominated causal effects, including large intervention, ablation, and mediation asymmetries. However, these causal effects do not correspond to geometric centrality: verbs do not occupy privileged positions under hyperbolic distance or norm-based criteria. We further show that common alignment metrics push representations toward the manifold boundary, obscuring geometric structure. Our results suggest that causal control in language models can be robustly non-linear and non-geometric, calling into question linearity-based interpretability assumptions.