AI Insight
This study investigates whether skin tone can be automatically estimated from dermoscopy images, a critical step toward auditing racial equity in dermatological AI systems. The researchers created MST-Derm, a benchmark of 500 ISIC 2018 dermoscopy images annotated by two independent raters using the Monk Skin Tone scale, finding that 60% of images were unrateable due to lesion occlusion of surrounding skin. Three pixel-based color estimation pipelines were tested and all produced statistically significant ordinal signal above chance, but performance remained limited, with the best method achieving a quadratic-weighted Cohen's Kappa of 0.43, largely constrained by dermoscopy's compressed color range and severe dataset bias toward lighter skin tones.
Why it matters
Public dermoscopy benchmarks currently lack skin tone labels, making it impossible to assess whether AI diagnostic tools perform equitably across diverse patient populations. This work provides foundational infrastructure and identifies key technical barriers that must be overcome before automated skin tone annotation can reliably support fairness auditing in clinical AI.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Skin-tone labels are absent from public dermoscopy benchmarks such as the International Skin Imaging Collaboration (ISIC), making it impossible to audit whether clinical AI performs equitably across skin tones. While several recent works estimate skin tone automatically from clinical photography and selfies, we ask whether this approach is feasible on dermoscopy, the primary imaging modality of these benchmarks. To answer this, we make three main contributions. First, we release MST-Derm, a dual-rater Monk Skin Tone (MST) annotation benchmark on 500 ISIC 2018 images. Raters were given an explicit unrateable option for crops where the skin surrounding the lesion was too occluded to label confidently. We find that 60% of images were marked unrateable, yielding a 193-image consensus subset (quadratic-weighted Cohen’s Kappa = 0.82). Second, we conduct a systematic feasibility study of three pixel-based MST annotation pipelines spanning the principal families in prior work: palette matching in perceptual colour space, robust colour statistics, and projection to a 1D colorimetric scalar. All three pipelines produce ordinal signal above chance (95% confidence intervals on quadratic-weighted Kappa exclude zero). However, ISIC 2018’s extreme light-skin bias leaves 82% of the evaluation set at MST 2, giving a constant "always predict MST 2" baseline an accuracy floor the methods cannot overcome. To separate algorithmic signal from dataset bias, we evaluate on a class-balanced subset. The best method reaches quadratic-weighted Kappa = 0.43 against the trivial baseline of Kappa = 0.00, confirming the signal is genuine. Third, we diagnose this performance ceiling. We trace the bottleneck to two causes: dermoscopy’s specialised illumination physically compresses the colour range on which lighter skin tones differ, and ISIC’s dataset skew makes standard absolute-accuracy metrics uninformative. We conclude that while pixel-based colour features carry real MST signal on dermoscopy, current performance is insufficient for autonomous annotation. We release the benchmark, annotation protocol, all prediction runs, and analysis code to facilitate the development of robust skin-tone estimators, a vital prerequisite for accurately auditing fairness and mitigating bias in dermatological machine learning.
Source: Pixel-Based Skin Tone Estimation on Dermoscopy: A Dual-Rater MST Benchmark and Feasibility Study