AI Insight
Researchers have developed MirrorCode, a new benchmark that tests AI systems' ability to reimplement entire software programs by observing only their behavior, without access to source code. The benchmark includes 25 diverse programs spanning areas like Unix utilities, bioinformatics, and cryptography, with the strongest current AI model achieving 56% success rate, including reimplementing a 16,000-line bioinformatics toolkit. However, completing larger tasks requires substantial computational resources, with some attempts costing $2,600 and taking 19 days.
Why it matters
This research demonstrates that AI has reached a capability threshold for autonomous software engineering on complex, long-horizon tasks when requirements are precisely specified. The findings suggest AI could significantly transform software development practices, though the high computational costs currently limit practical deployment for the most complex reimplementation tasks.
Understand the Science
arXiv:2606.30182v2 Announce Type: replace
Abstract: AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit – a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, $2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.
Source: MirrorCode: AI can rebuild entire programs from behavior alone