AI Insight
This study examines how large language models (LLMs) handle reading (input processing) and writing (output generation), comparing them to the human brain where these functions can fail independently through separate neural pathways. The researchers found that in decoder-only LLMs like GPT-2, OPT, and Pythia, the input embedding matrix and output projection matrix are partially entangled but not identical, with the output code drifting approximately 3.2 times farther than the input code during training. However, they identify critical methodological flaws in standard behavioral tests that claim to measure reading-writing dissociation in LLMs, arguing these tests cannot actually isolate comprehension from production due to the fundamental architecture of next-token prediction models.
Why it matters
This research challenges common assumptions about how LLMs process language and reveals limitations in current methods for evaluating their capabilities. The findings suggest that LLMs fundamentally differ from human cognitive architecture, which has implications for AI safety research, model interpretability, and understanding the boundaries of what these systems can achieve.
Understand the Science
arXiv:2607.24797v2 Announce Type: replace
Abstract: In the literate human brain, reading and writing doubly dissociate: a ventral decoding route (pure alexia) and a fronto-parietal encoding route (pure agraphia), sharing a partial orthographic core. A decoder-only large language model (LLM) drives both from one autoregressive path optimized on text (a emph{cultural} invention, not an evolved instinct). We ask how entangled it is, comparing an input-side “reading code” $mathbf{W}_{E}$ with an output-side “writing code” $mathbf{W}_{U}$ via an index $mathcal{E}in[0,1]$ (CKA, Procrustes residual, mutual $k$-NN) calibrated against an independent-init floor and tied ceiling. On GPT-2, OPT and Pythia (14M–1.4B), untied models hold one emph{coupled but sub-ceiling} code ($mathcal{E}=0.23$–$0.35$, far above floor) on a non-monotonic couple-then-differentiate trajectory, $mathbf{W}_{U}$ drifting $sim$3.2$times$ farther than $mathbf{W}_{E}$ in every decile. Equally informative is a negative: the matching behavioural test, that comprehension and production fail together rather than dissociate, cannot be run. For minimal pairs the alexia analogue is empty by theorem: greedy production implies a vocabulary-wide argmax, so it wins the pairwise ranking. Differential-damage indices are not scale-identified: heavy-tailed damage makes linear standardizations collapse onto their larger term, and the rank transform fixing this is bounded, so its null saturates. Both scores also contain the target’s log-probability, which alone explains most of their variance and manufactures the apparent coupling. We withdraw a coupling statistic, a cross-level bridge and a separation measure. In a model reading and writing off one next-token distribution, no output-side pair isolates either ability: entanglement needing no index to see. By analogy, not homology, this situates LLMs in the space of possible minds.