AI Insight
This study examines how to effectively use codebooks designed for human text classification as prompts for large language models. Testing across three experimental economics classification tasks and multiple LLMs, researchers found that codebooks achieved 82-88% agreement with human annotators, with prompt content detail being as important as model choice for complex tasks. Larger models better utilize detailed information and are more resilient to prompt variations, while smaller models are more sensitive to how information is presented and can be negatively affected by excessive detail.
Why it matters
The findings suggest researchers should prioritize creating detailed, human-readable classification instructions rather than focusing on prompt engineering tricks like special formatting or reasoning chains. This approach makes LLM-based text classification more accessible and reliable for researchers who already have established coding protocols.
by Can Çelebi, Stefan P. Penczynski
Researchers who classify text with large language models often possess codebooks written for human annotators. We ask how these codebooks can serve as prompts. Using three codebooks from experimental economics (one on promise classification, two on strategic thinking), we vary the level of information in the prompt, then vary its formatting, framing and wording at the lowest and highest information levels, across two proprietary and two open-weight models. Used as prompts, the codebooks reach 82–88% agreement with human annotators across the three tasks. On the recognition-heavy task (promise classification), model choice accounts for most of the variation in accuracy; on the learning-heavy tasks (strategic thinking classification), the level of detail in the classification instructions carries comparable weight. These information components partly substitute for one another, whereas model reasoning does not reliably compensate for missing content and yields little or no improvement once the content is present. Larger models make better use of additional information and are more robust to formatting, framing, and wording of the prompt, while smaller models can be hurt by extra information and are more sensitive to how the information is presented. Our results advocate for a shift in focus from prompt engineering techniques (formatting, framing, reasoning, etc.) to the content of the prompt: preparing instructions as one would for human annotators, with detailed context, category definitions, and examples.
Source: Much Ado about Prompting: LLM classification of text messages from experiments