AI Insight
Researchers tested whether a transformer neural network (GPT-2 architecture) could learn patterns in a mathematical sequence where each natural number is represented as a tree structure based on its prime factorization. The model, trained on the first 100 billion elements, achieved approximately 40% word accuracy in prediction tasks and maintained stable performance when tested on elements far beyond its training range (at positions 10^13 to 10^15). The trained model successfully distinguished the authentic arithmetic sequence from control sequences that preserved token frequencies but not their mathematical structure, demonstrating that it captured genuine arithmetic regularities rather than merely memorizing surface-level statistics.
Why it matters
This work provides empirical evidence about transformer networks' ability to learn structured mathematical patterns from data alone, which has implications for understanding the capabilities and limitations of large language models in mathematical reasoning. The findings suggest these architectures can extract non-trivial regularities from deterministic sequences, offering insights for both AI development and computational approaches to number theory.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We study whether a transformer network can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers. Each integer is mapped into a rooted planar tree and the resulting sequence $mathbb{N}mathscr{T}$ defines an arithmetic text with measurable statistical structure. A transformer network (the GPT-2 architecture) is trained from scratch on the first $10^{11}$ elements and evaluated on next-token prediction and masked-token prediction tasks, with a Hidden Markov Model as baseline and a scaling analysis over context window, dataset size, vocabulary size and model size. The model reaches a word accuracy of about $0.4$, well above the baseline, and its performance remains stable on test blocks located at $10^{13}$-$10^{15}$, far beyond the training interval. Moreover, the likelihood assigned by the trained model separates the arithmetic text from two controls: synthetic token sequences that preserve the marginal token frequencies but not their order, with a separation that widens with the context, and sequences containing more than three consecutive square-free integers, a configuration that arithmetic forbids. These results indicate that the transformer captures regularities of the arithmetic text that go beyond its frequency profile.
Source: Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees