ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
Rattachement africain : cn, us. Niveau de preuve : code pays fourni par la source.
Le résumé fourni par la source
Offline reinforcement learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, adaptive conservative level in Q-learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used datasets for deep data-driven reinforcement learning (D4RL) benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared with existing offline DRL baselines.
Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.
Le contrôle bibliographique ouvert
DOI retrouvé dans Crossref DOI retrouvé, mais le titre doit être comparé manuellement.
- Titre Crossref
- ACL-QL: Adaptive Conservative Level in <i>Q</i>-Learning for Offline Reinforcement Learning
- Date Crossref
- 01/06/2025
- Éditeur
- Institute of Electrical and Electronics Engineers (IEEE)
- Type
- journal-article
Ce recoupement confirme des métadonnées liées au DOI. Il ne confirme ni la méthode ni les conclusions de l’étude, et il ne compte pas comme une seconde source scientifique indépendante.
Où se fait cette recherche
-
Beijing Institute of Technology pays non établi dans la noticeUniversité ou école supérieure
-
SAIC-GM (China) pays non établi dans la noticeEntreprise
-
Syracuse University Department of Electrical Engineering and Computer Science pays non établi dans la noticeUniversité ou école supérieure
-
Beijing Advanced Sciences and Innovation Center pays non établi dans la noticeStructure de recherche
-
Midea Group (China) pays non établi dans la noticeEntreprise
-
Shanghai201700 pays non établi dans la noticeInstitution
-
with Beijing Innovation Center of Humanoid Robotics pays non établi dans la noticeInstitution
Beijing Institute of Technology, SAIC-GM (China) et Department of Electrical Engineering and Computer Science — Syracuse University, avec 4 autres affiliations.
Une affiliation ne permet pas de déduire la nationalité d’un auteur.