AI Insight
This study investigated whether labeling answers as coming from "ChatGPT" versus a generic "computer" influenced how 103 participants evaluated responses to reasoning tasks. Participants accepted ChatGPT-labeled answers more often overall (68.0% vs 63.4%), with the strongest effect appearing in simple arithmetic problems rather than in cognitive reflection test questions designed to elicit intuitive but incorrect responses. The source label did not significantly affect participants' ability to distinguish between correct and incorrect answers on reasoning tasks, suggesting the ChatGPT cue increased general acceptance without improving judgment accuracy.
Why it matters
As AI systems like ChatGPT become common sources of information, understanding that their branding may increase uncritical acceptance of answers—particularly for straightforward questions—has implications for education, decision-making, and the design of AI interfaces. This highlights potential risks of over-reliance on AI-attributed information without corresponding improvements in critical evaluation.
Understand the Science
IntroductionAs generative AI systems increasingly provide answers in everyday and academic contexts, understanding how source attribution influences evaluative judgment has become critical.MethodsIn a within-subjects design, 103 participants evaluated 30 pre-generated reflective-correct, intuitive-incorrect, and arithmetic control-correct answers labeled ChatGPT or computer. Assignment of two parallel batteries to source labels and trial presentation order were randomized, and displays were visually matched. An independent unlabeled pretest with 35 participants supported battery comparability.ResultsParticipant-level analysis showed greater overall acceptance under ChatGPT (68.0% versus 63.4%, p = 0.002). A trial-level logistic mixed model with crossed participant and parallel-problem-pair random intercepts showed a source-by-category interaction (p < 0.001): acceptance was higher under ChatGPT for arithmetic controls (OR = 6.08, 95% CI [2.70, 13.69]), while source contrasts within the two CRT categories were not significant. Judgment accuracy is reported descriptively. Signal-detection analyses of CRT answers detected no significant source difference in sensitivity (p = 0.930) or criterion (p = 0.082). Evaluation times were longer under ChatGPT in the raw-time analysis (p = 0.034), but the source difference was not significant after log transformation (p = 0.180). Exploratory correlations detected no significant associations between source-related differences and trust, AI attitudes, or critical-thinking disposition.DiscussionThese findings distinguish overall source-related endorsement, particularly for arithmetic controls, from successful discrimination between correct and incorrect CRT answers.
Source: The ChatGPT source cue shapes answer acceptance in a reasoning task