AI Insight
KernelGenBench is a new benchmark designed to test how well large language models and AI agents can automatically generate specialized computing kernels across different hardware platforms and operator sources. Testing 210 operators from PyTorch, vLLM, and cuBLAS across six hardware platforms, the research found that AI-generated kernel quality varies dramatically depending on the source and hardware, with agent-based methods improving correctness but requiring millions of tokens per successful operator. For example, AutoKernel accuracy dropped from 87% on NVIDIA hardware to 25% on Iluvatar CoreX, demonstrating that performance in one setting does not reliably predict success in others.
Why it matters
This research reveals critical limitations in using LLMs to automate kernel development for AI accelerators, showing that successful code generation on familiar hardware does not guarantee deployment readiness across diverse platforms. The findings suggest organizations cannot rely on single-platform benchmarks to evaluate LLM capabilities for production kernel generation, which has important implications for AI infrastructure development costs and timelines.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Modern AI systems depend on specialized accelerator kernels, whose development is complicated by increasingly diverse operators and hardware. LLMs and agentic systems promise to automate this work, but existing evaluations do not show whether their performance transfers across operator sources and hardware platforms, or what such transfer costs. We present KernelGenBench, the first unified multi-source and multi-chip infrastructure for evaluating LLM- and agent-generated Triton kernels. With a common Triton target spanning six hardware platforms, it provides the broadest cross-vendor hardware coverage among existing kernel-generation benchmarks. We report two controlled analytical views: KernelGenBench-MS (Multi-Source) covers 210 operators from PyTorch ATen, production vLLM operators, and proprietary cuBLAS routines, while KernelGenBench-MC (Multi-Chip) evaluates a semantically stable 110-operator subset across six hardware platforms. Our evaluation consumed over 15 billion tokens. Agentic execution improved correctness, but no method dominated across sources and platforms: vLLM posed the strongest correctness challenge, cuBLAS set the highest performance ceiling, and AutoKernel accuracy fell from 87% on NVIDIA to 25% on Iluvatar CoreX. These improvements were costly: specialized agents averaged 4.99 million tokens per successful operator, rising to 6.25 million for CUDA Optimized Skill. The results establish operator source, hardware platform, and agentic scaffold as distinct dimensions of kernel-generation capability, and show that success in a familiar source-hardware setting is not a reliable proxy for deployment readiness.
Source: KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation