AI Insight
This study examines the trade-off between reducing carbon emissions and maintaining service quality in geographically distributed Large Language Model (LLM) inference systems. Researchers developed a sophisticated simulator calibrated with real hardware measurements to test various carbon-aware scheduling strategies that route AI inference requests across datacenters based on local grid carbon intensity. Their most advanced scheduler, using facility-level mixed-integer linear programming, achieved up to 60% carbon emission reductions compared to standard round-robin scheduling while maintaining service-level objectives with only 3% average violation rate across three different LLMs.
Why it matters
As LLM usage rapidly expands, datacenter carbon emissions from AI inference are becoming a significant environmental concern. This research demonstrates that intelligent request routing across datacenters can substantially reduce the carbon footprint of AI services without severely compromising user experience, providing a practical pathway for more sustainable AI deployment.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions. Although carbon-aware scheduling can reduce these emissions in geo-distributed datacenters by exploiting the spatiotemporal variations in grid carbon intensity, the latency-sensitive nature of LLM serving constrains this design space. While prior studies investigate the carbon savings achievable through spatial scheduling, they oversimplify the dynamics of modern inference engines, such as queueing and continuous batching, which strongly shape inference latency. We address this issue by designing and evaluating a latency-centric, geo-distributed inference simulator, calibrated with hardware-grounded latency and power measurements, that reproduces these dynamics and enables the study of carbon savings under strict service-level objectives (SLOs). We then formulate a series of SLO- and carbon-aware schedulers of increasing sophistication, from per-request heuristics to batch-level optimization: (i) a greedy heuristic, (ii) an integer linear programming (ILP)-based scheduler jointly assigning batches of requests, (iii) an iterative ILP-based solution projecting its own assignments back into the load estimates to capture load coupling, and (iv) a facility mixed-integer linear programming (MILP) scheduler pricing the energy of batched requests collectively rather than in isolation. Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon-SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy, with an average SLO violation of 3%. Our characterization further shows that savings grow with the spread of carbon intensity across datacenters and shrink as system load increases, while SLO attainment depends on how frequently schedulers receive datacenters’ load updates.
Source: Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving