AI Insight
This paper introduces Geometric Zonotope Soft Actor-Critic (GeZo-SAC), a reinforcement learning algorithm that improves upon standard actor-critic methods by using geometric representations called zonotopes to adaptively control how pessimistic the algorithm is about value estimates. Instead of simply taking the minimum of two critic networks everywhere, GeZo-SAC uses the geometric properties of zonotopes to measure disagreement between critics and adjusts the aggregation method accordingly, blending between a weighted average and minimum based on the level of disagreement. Testing on four MuJoCo locomotion tasks showed that GeZo-SAC achieved the highest performance on two benchmarks while maintaining competitive results on the others, with the lowest energy consumption per distance traveled and minimal overestimation bias.
Why it matters
This approach could lead to more energy-efficient robot locomotion controllers by reducing unnecessary actuator work while maintaining performance. The method's ability to deploy standard policies at inference time while benefiting from geometric representations during training makes it practically applicable to real robotic systems without additional computational overhead.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Actor–critic methods in reinforcement learning control overestimation bias by taking the minimum of two critics. This uses the same aggregation rule everywhere, regardless of how the critics disagree. We propose Geometric Zonotope Soft Actor-Critic (GeZo-SAC), which uses auxiliary geometric representations to adapt critic pessimism to the state and action. Alongside its scalar value, each critic predicts a set of generators defining a zonotope. Probing this zonotope along sampled directions provides a geometric width, subtracted from each critic value as a pessimistic offset, and a measure of disagreement between the two critics, aggregated with log-sum-exp as a smooth approximation to the tropical (max-plus) addition. This disagreement controls how the critics are combined, moving from a width-weighted average toward the usual minimum as disagreement increases. At inference, the deployed policy is an unmodified SAC actor, since the generators are used only on the critic side during training. Across four MuJoCo-v5 locomotion benchmarks and six off-policy baselines, GeZo-SAC achieves the highest mean return on Ant-v5 and Hopper-v5 and remains competitive with other methods on the remaining tasks. Our analysis further shows that GeZo-SAC achieves the lowest average actuator work and action effort per metre among the evaluated methods, while maintaining near-zero measured overestimation frequency across all four environments.
Source: A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning