AI Insight
This paper introduces CriterAlign, a new framework for evaluating code-generation systems that makes pairwise comparisons at the criterion level rather than comparing aggregate scores from independent evaluations. The system incorporates Human-Preference-Aligned Guidance (HPAG), which learns from discrepancies between human preferences and automated judge predictions, and uses techniques like tie-driven refinement and swap-consistency filtering. Testing on the BigCodeReward dataset showed the approach improved accuracy from 60.4% to 66.3% when applied to a Qwen2.5-VL-32B model.
Why it matters
As AI-generated code becomes more prevalent in software development, having reliable automated evaluation systems that align with human judgment is crucial for assessing code quality beyond simple correctness. This framework could improve the development and deployment of code-generation tools by providing more accurate, interpretable quality assessments.
Understand the Science
-cross
Abstract: Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.
Source: CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging