AI Insight
This research introduces a semantic inequivalence game (SInQ) to generate synthetic training data for improving Large Language Models' understanding of program semantics. The method uses two AI agents that work semi-adversarially: one generates semantically different program variants while the other identifies inputs that expose these differences. Testing on multiple benchmarks showed substantial improvements, including cross-language vulnerability detection where models trained only on Python code successfully detected vulnerabilities in C/C++ code.
Why it matters
This approach addresses a critical weakness in current LLMs by generating training data for complex code reasoning tasks that are difficult to obtain naturally. The demonstrated cross-language transfer and improved vulnerability detection could enhance software security tools and code analysis systems without requiring extensive language-specific training data.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics. Finding training examples to teach LLMs to solve these tasks can be challenging.
In this work, we explore a method to synthetically generate code reasoning training data based on a semantic inequivalence game (SInQ): a generator agent creates program variants that are semantically distinct, derived from a dataset of real-world programming tasks, while an evaluator agent has to identify input examples for which they behave differently. The agents train each other semi-adversarially, improving their ability to understand the underlying logic of code.
We evaluated our approach on multiple code generation and understanding benchmarks, including cross-language vulnerability detection (Lu et al., 2021),, where our method improves vulnerability detection in C/C++ code despite being trained exclusively on Python code, and the challenging Python builtin identifier swap benchmark (Miceli Barone et al., 2023),, showing that whereas modern LLMs still struggle with this benchmark, our approach yields substantial improvements.
We release the code needed to replicate the experiments, as well as the generated synthetic data, which can be used to fine-tune LLMs.
Source: Program Semantic Inequivalence Game with Large Language Models