AI Insight
MultiHashFormer is a new language model architecture that represents tokens as unique sequences of hash IDs rather than individual embedding vectors, allowing vocabulary size to scale without proportionally increasing parameters. The system uses multiple hash functions to create collision-free signatures, which are processed through a Hash Encoder-Decoder framework alongside a standard Transformer. Testing at 100M, 1B, and 3B parameter scales shows MultiHashFormer outperforms standard Transformer language models while maintaining constant parameter footprint even when expanding to multilingual vocabularies.
Why it matters
This approach could significantly reduce the computational resources and memory required to train and deploy large language models, especially for applications involving multiple languages or large vocabularies. The constant parameter footprint regardless of vocabulary size makes it particularly valuable for resource-constrained environments and multilingual AI systems.
Understand the Science
Abstract: Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.
Source: MultiHashFormer: Hash-based Generative Language Models