AI Insight
Researchers developed Semantic Feature Analysis (SFA), a method that improves AI agent performance by analyzing existing execution traces rather than conducting expensive trial-and-error searches. SFA extracts semantic patterns from past agent behaviors, identifies which features correlate with successful outcomes using decision trees, and automatically injects corrective instructions into agent prompts. When tested against five existing optimization methods across four benchmarks, SFA consistently improved agent accuracy while avoiding the high computational costs of traditional prompt optimization approaches.
Why it matters
This approach significantly reduces the cost of improving AI agents that use expensive external tools or APIs, where traditional optimization methods can spend hundreds of dollars testing candidate prompts. SFA enables practical agent improvement in budget-constrained scenarios where existing optimizers often fail to find any improvements at all.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Ambiguity is an inherent property of natural-language agent specifications. When a system prompt leaves behaviour underdetermined, identical inputs follow divergent execution paths and produce inconsistent outcomes. The standard remedy is prompt optimisation: propose candidate prompts, run the agent to score them, and keep the best. This loop pays for the agent twice: once to generate candidates and again to rank them. On a tool-using agent whose rollouts cost dollars and minutes, the ranking cost dominates and budget-constrained optimisers routinely fail to find improvements.
We present Semantic Feature Analysis (SFA), a pipeline that repairs agent specifications without running any search. SFA reads execution traces the agent has already produced, clusters the outputs of each workflow node, decomposes them into semantic feature classes using an extended subject-verb-object schema, ranks those features by their contribution to outcome separation using a decision tree, and injects the surviving features as corrective statements into the affected node’s system prompt. Because it never ranks candidate prompts, it never spends a rollout on selection.
We evaluate SFA against five prompt optimisers (GEPA, MIPROv2, SIMBA, BootstrapFewShot with random search, InferRules) and a single-reflection control, budget-matched in dollars at three budget levels across four benchmarks (IF-Bench, HotpotQA, HoVer, and GAIA). SFA consistently improves over the unmodified agent across benchmarks and budget levels, with the largest gains where rollouts are most expensive. On GAIA, where budget-constrained optimisers cannot afford to score even one candidate, SFA improves accuracy while other arms return their seed unchanged.
Source: Semantic Feature Analysis: Improving Agents Without Searching Over Rollouts