AI Insight
LatentPress introduces a novel method for compressing conversational histories and long documents into continuous memory tokens that language models can read directly, bypassing the need for text or image reconstruction. The system achieves 4-16x compression while training only small adapter modules (0.1% of decoder size), and demonstrates superior performance on memory and question-answering benchmarks compared to text summaries and OCR-based approaches. The method processes conversations in 43ms and enables 5-9x faster reading than uncompressed contexts while maintaining or exceeding accuracy at moderate compression ratios.
Why it matters
This approach could significantly reduce computational costs and latency in AI systems that process long contexts, such as chatbots with extended conversation histories or document analysis tools. By creating a machine-native representation that bypasses human-readable formats, LatentPress offers a more efficient alternative to current context-handling methods in large language models.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference. A small reader-matched writer compresses $4$-$16times$ while training only an adapter (4.2M-26.2M parameters, $sim!0.1%$ of the decoder). On LongMemEval, LatentPress reaches $0.504$ accuracy at $7.70times$ compression versus $0.490$ for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at $4$-$8times$ compression, while $16times$ trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is $5$-$9times$ faster than raw context or cached OCR. We validate the interface under two transfer settings, zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived QA to unseen LongBench document domains, establishing direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation of the experiments could be found at: https://github.com/HJSang/LatentPress .
Source: LatentPress: Context Compression Beyond Text and Vision