AI Insight
This paper addresses safe offline reinforcement learning when supervisors only provide binary feedback indicating the first unsafe action in a trajectory, rather than detailed cost labels for every step. The authors propose Redistribution-based Cost Inference (RCI), which converts this sparse feedback into detailed per-step cost estimates through return decomposition, enabling standard constrained offline RL methods to learn safer policies. Experiments on highway driving and robotic manipulation tasks show that RCI substantially reduces safety violations compared to baseline methods that use sparse signals directly or rely on classifier-based cost inference.
Why it matters
This work makes safe reinforcement learning more practical by reducing the annotation burden on human supervisors, who can provide simple stop signals rather than detailed safety labels. The approach could enable safer deployment of autonomous systems in domains like robotics and autonomous driving where obtaining dense safety annotations is expensive or infeasible.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.
Source: Redistribution-based Cost Inference Improves Sparse Safe Offline RL