AI Insight
This paper presents new methods for group distributionally robust optimization in decentralized networks, where multiple workers collaborate to train machine learning models that perform well across different data groups without a central coordinator. The researchers developed algorithms that directly optimize the original objective function and provide the first theoretical convergence guarantees for individual worker models in both semi-decentralized and fully decentralized settings. Their approach achieves a convergence rate of O(sqrt((log m)/T)) over T update rounds and demonstrates improved worst-case performance across workers in experiments.
Why it matters
This work addresses critical challenges in distributed machine learning for applications requiring fairness and robustness across different population groups, such as healthcare diagnostics or financial systems. The decentralized approach enables collaborative learning while preserving data privacy and reducing reliance on central servers, making it practical for federated learning scenarios where data cannot be centrally aggregated.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We study group distributionally robust optimization (GDRO) over decentralized networks, where workers collaborate to learn robust models by minimizing the maximum local risk. Existing methods optimize only dual-regularized surrogates and establish guarantees for a globally averaged model that is not available to workers. In this paper, we directly optimize the original GDRO objective and establish guarantees for local models in semi-decentralized (SemiDec) and fully decentralized (Dec) settings. A key difficulty is that applying the primal-dual decomposition from centralized GDRO yields worker-dependent losses whose evaluation requires unavailable cross-worker information. To address this challenge, we develop a novel error decomposition that recasts decentralized GDRO as (i) a decentralized online convex optimization (D-OCO) problem and a prediction-with-expert-advice problem in SemiDec, and as (ii) two coupled D-OCO problems in Dec, with discrepancy terms capturing inconsistencies among local decisions. By isolating consensus errors in analysis, we develop modular frameworks that support different consensus routines. We then establish the emph{first} per-worker convergence guarantees for optimizing the original GDRO objective over decentralized networks. Instantiations with accelerated gossip and Choco-gossip achieve an $O(sqrt{(log m)/T})$ convergence rate over $T$ update rounds, while experiments demonstrate improved worst-worker performance over baselines.
Source: Group Distributionally Robust Optimization over Decentralized Networks