AI Insight
This paper introduces Bevy, a new system for running large mixture-of-experts (MoE) language models on local hardware by improving how AI experts are loaded from storage to GPU memory. Rather than predicting which experts will be needed next (prefetching), Bevy allows multiple inference requests to proceed at different speeds, letting requests with available experts run ahead while others wait, then batching requests together when efficient. Across testing platforms, Bevy achieved up to 3.46 times higher throughput compared to existing systems by overlapping computation with data fetching without requiring prediction.
Why it matters
This work could enable more efficient deployment of large MoE models on consumer hardware with limited GPU memory, making advanced AI models more accessible without requiring expensive cloud infrastructure. The scheduling approach may generalize to other resource-constrained AI inference scenarios where data movement is a bottleneck.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Expert offloading enables large mixture-of-experts (MoE) language models to run locally, but conventional batching stalls the GPU while missing experts are fetched. A popular remedy, known as prefetching, is to predict what experts will be needed soon and fetch them using idle link bandwidth. Imprecise prediction, however, wastes bandwidth and GPU memory on fetching unnecessary experts. Our key finding is that prefetching can be harmful when there is an expert cache in GPU memory. This is because as the cache includes more necessary experts, expert prediction becomes less precise. Our key insight is that proper request scheduling beats prefetching. Building on this insight, we present Bevy, an MoE inference system based on runahead inference for concurrent requests. A request whose experts are available can run ahead to later layers while other requests wait for fetches, revealing the experts it needs next. Bevy can then fetch the missing experts so identified while computing ready requests, overlapping computation and fetching without predicting expert selections. Running requests separately, however, repeats reads of shared weights in GPU memory and increases compute time. To limit this cost, Bevy’s self-balancing scheduler batches ready requests at each layer and prioritizes lagging requests, adapting their grouping as expert availability changes. Across two local platforms, Bevy delivers up to 3.46$times$ the decode throughput of llama.cpp and up to 1.55$times$ that of the fastest prefetching baseline.
Source: Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests