AI Insight
This study examines how different detection methods respond to data corruption in federated learning systems. Researchers found that prediction-label loss effectively detects persistent label flips (AUC 0.85-0.95), while uncertainty measures remain at chance level (0.49-0.50). However, for image noise corruption, entropy-based uncertainty measures perform comparably to loss-based detection (0.64-0.67 AUC), demonstrating that no single signal reliably detects all corruption types.
Why it matters
These findings have important implications for building robust federated learning systems, particularly in medical imaging, autonomous vehicles, and other applications where data quality directly affects model reliability. The research suggests that federated learning systems should employ multiple corruption-detection strategies tailored to specific corruption types rather than relying on a single detection method.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Federated learning (FL) data corruption can affect either inputs or labels, but it remains unclear whether input-conditional uncertainty and prediction-label loss expose these corruption modes equally. This paper compares two corruption-detection signals in FL: input-conditional uncertainty and prediction-label loss. The uncertainty signal is characterised using a learned aleatoric variance estimate together with Monte Carlo (MC) dropout variance and entropy measures, while the loss is computed against the supplied label. We test these signals against additive image noise and persistent random label flips. On ResNet-20 with CIFAR-10 and SVHN under Dirichlet partitions with data that are not independent and identically distributed (non-IID), the two corruption types behave differently. For persistent random label flips, the within-client per-sample area under the receiver operating characteristic curve (AUC) is 0.85 on CIFAR-10 and 0.95 on SVHN for prediction-label loss, while every uncertainty estimator stays at chance (0.49–0.50). This pattern is consistent with the model remaining confident in the underlying image despite the supplied label being wrong. For image noise, expected-entropy uncertainty rises above chance (0.67 on CIFAR-10 and 0.66 on SVHN), while loss responds comparably (0.64 on both). Each signal is therefore the stronger detector for a different corruption: the prediction-label loss for persistent label flips, and expected-entropy uncertainty for image noise, with its advantage becoming apparent as federation-wide corruption prevalence increases. Robust FL data-quality assessment should match the signal to the corruption rather than rely on uncertainty alone across corruption types.
Source: Different Corruptions, Different Signals: Uncertainty and Loss in Federated Data Quality