AI Insight
This study examines Classifier-based Quality Filtering (CQF), a method used to filter large web-crawled datasets for training AI models by scoring documents based on their similarity to high-quality reference data. Researchers found that training on CQF-filtered data can produce better results than training directly on high-quality datasets alone, because CQF effectively removes problematic examples from both low-quality and high-quality data sources. The work introduces a new optimization-based definition of data quality that can be estimated through small-scale experiments, revealing that effective filtering primarily removes harmful data rather than identifying perfect examples.
Why it matters
These findings challenge the current trend of investing heavily in collecting more high-quality training data, suggesting that better filtering of existing mixed-quality datasets may be more cost-effective. The research provides practical guidance for organizations training large AI models on how to improve performance through smarter data selection rather than simply acquiring more premium data.
Understand the Science
arXiv:2510.00866v4 Announce Type: replace
Abstract: Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier’s score and retains only the top-scoring ones. We provide an in-depth analysis of CQF.
We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality set. Importantly, we find that training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large. This finding alone is particularly striking, given the substantial effort and cost recently devoted to augmenting high-quality data. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well as the low-quality one. Finally, we introduce an optimization-driven notion of data quality and demonstrate that it can be reliably estimated using small-scale proxy experiments. Altogether, our results both elucidate the mechanisms behind CQF and deepen our understanding of data selection methods widely used in practice.
Source: Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining