AI Insight
This paper introduces OLAS, a new active learning framework that addresses the problem of noisy labels by strategically assigning samples to labelers and selecting which samples to label based on both labeler expertise and model uncertainty. The method uses optimization formulations to minimize labeling errors, with theoretical analysis providing closed-form solutions under certain conditions. Testing on benchmark datasets and a real-world warranty claim classification task demonstrates that OLAS achieves superior or comparable classification accuracy to existing active learning strategies while using only one label per sample.
Why it matters
This work addresses a critical practical challenge in machine learning applications where labeling is expensive and label quality varies across annotators. The framework could reduce costs and improve model performance in domains like medical diagnosis, content moderation, or any field requiring expert human annotation with variable expertise levels.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.
Source: Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection