AI Insight
This study investigates asymmetric confusion patterns in visual categorization tasks, comparing how humans and deep neural network vision models make errors when classifying natural images under 12 types of perturbation. The researchers find that humans display broad but relatively weak directional asymmetries across many category pairs, while machine vision models tend to collapse predictions into a small number of dominant categories with stronger asymmetries. Using rate-distortion geometry to analyze these patterns, the authors demonstrate that these two distinct error structures represent fundamentally different generalization strategies, even when overall accuracy is matched, and that adversarial robustness training reduces asymmetry magnitude without reproducing the distributed human-like error structure.
Why it matters
These findings suggest that accuracy alone is an insufficient benchmark for evaluating how closely machine vision systems replicate human-like perception, which has direct implications for the design of AI systems intended to operate in human-centered contexts or to serve as models of biological vision. Identifying directional confusion structure as a measurable signature of inductive bias could lead to better diagnostic tools for assessing alignment between human and machine perception.
arXiv:2604.21909v2 Announce Type: replace-cross
Abstract: To humans, a robin seems more like a bird than a bird seems like a robin, but does this asymmetry also hold for machine vision? Humans and modern vision models can match each other in accuracy while making systematically different kinds of errors, differing not in how often they fail, but in who gets mistaken for whom. We show these directional confusions reveal distinct inductive biases invisible to accuracy alone. Using matched human and deep neural network responses on a natural-image categorization task under 12 perturbation types, we quantify asymmetry in confusion matrices and link its organization to the geometry of the information–error trade-off – how efficiently, and how gracefully, a system generalizes under distortion. We find that humans exhibit broad but weak asymmetries across many class pairs, whereas deep vision models show sparser, stronger directional collapses into a few dominant categories. Robustness training reduces overall asymmetry magnitude but fails to recover this human-like distributed structure. Generative simulations further show that these two asymmetry organizations shift the trade-off geometry in opposite directions even at matched accuracy, explaining why the same scalar asymmetry score can reflect fundamentally different generalization strategies. Together, these results establish directional confusion structure as a sensitive, interpretable signature of inductive bias that accuracy-based evaluation cannot recover.