AI Insight
This study investigates whether the sentiment of topics in political news articles causally influences how readers perceive their ideological stance, comparing human expert annotations with large language model (LLM) assessments. Using articles from AllSides and sentiment data from Llama-3.3-70b, researchers applied causal inference methods across four annotation paradigms and found that zero-shot LLMs systematically overestimate effect sizes compared to human annotators, while fine-tuned models produce estimates closer to human-generated labels. The findings suggest LLMs can reliably detect the presence and direction of ideological effects but struggle with accurately measuring their magnitude.
Why it matters
This research has important implications for researchers increasingly using LLM-generated annotations as substitutes for human judgment in social science and political communication studies. The finding that LLMs misestimate effect magnitudes suggests caution is needed when using AI-generated labels for causal analyses, particularly in sensitive domains like political ideology measurement where accurate effect sizes matter for understanding media influence and polarization.
Understand the Science
arXiv:2606.06715v2 Announce Type: replace-cross
Abstract: We ask whether topic sentiment has a causal effect on perceived political ideology, and whether the answer depends on who assigns the ideology label. Using articles from AllSides, paired with shared sentiment annotations from Llama-3.3-70b-versatile, we compare ideology labels from expert human annotators, GPT-4o-mini (baseline and finetuned), and Llama-3.3-70B. We apply Double Machine Learning (DML) and mediation analysis across all four annotation paradigms. Zero-shot LLMs regularly inflate effect sizes relative to human annotations, while fine-tuning often attenuates them back toward the human scale. Our results have implications for the use of LLM annotations as silver labels and as proxies for human judgment in downstream causal analyses: they may be reliable for recovering the presence and direction of effects on the partisan topics, but not their magnitude, leading to over- or under-prediction of some ideology given particular topics.