AI Insight
A randomized experiment with 164 law students found that untrained access to large language models actually harmed performance on legal analysis tasks, with users writing shorter answers and making more errors. However, students who received brief training before using the LLM scored significantly higher (0.27 grade points) than untrained users and adopted the technology more frequently. The benefits appear to stem primarily from increased adoption rates among trained users rather than improved effectiveness during use.
Why it matters
These findings suggest that simply providing access to generative AI tools in professional settings may be counterproductive without proper training. The research challenges assumptions that AI automatically benefits less-skilled workers and demonstrates that realizing productivity gains from generative AI requires deliberate investment in user education, not just technology deployment.
arXiv:2603.04982v3 Announce Type: replace-cross
Abstract: Can targeted user training unlock the productive potential of generative artificial intelligence in professional settings? We study this question using a randomized experiment in which 164 law students completed an issue-spotting examination under one of three conditions: no GenAI access, optional access to a large language model (LLM), or LLM access with a brief training intervention.
Untrained LLM access proved counterproductive: relative to participants without any LLM access, untrained users wrote significantly shorter answers, committed more case misstatements, and scored marginally lower, though most differences fall short of conventional significance. Training reversed this pattern. Trained participants adopted the LLM at higher rates (41% vs. 26%; p = 0.044), scored 0.27 grade points higher than untrained users–roughly one fine grade–(p = 0.027), and stated applicable rules more accurately (p = 0.014).
Principal stratification analysis suggests training operates primarily through adoption rather than effectiveness–the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance–though confidence intervals are wide.
More broadly, these findings challenge the view that GenAI primarily benefits lower-skilled workers: without training, higher-ability practitioners opt out while lower-ability users adopt but unproductively. Realizing GenAI’s productivity gains requires investment in both access and instruction.
Source: Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis