Attention With System Entropy for Optimizing Credit Assignment in Cooperative Multi-Agent Reinforcement Learning

Le résumé fourni par la source

In cooperative multi-agent reinforcement learning (MARL), value function factorization methods have been proposed to address the dimensionality explosion problem encountered in centralized training and decentralized execution (CTDE). The existing value function factorization methods lack a perspective from the global system when addressing credit assignment, failing to measure each agent’s contribution to the current system. A typical limitation is that these factorization approaches primarily rely on the proximity of local individual groups to assign credit. There are challenges such as the sensitivity of credit assignment weights to local information and instability in the learning process. In this study, we focus on redistributing agents’ features during the system evolution and suggest employing an enhanced attention mechanism with system entropy measure to factorize value function. Specifically, this method emphasizes each agent’s representation between their local and global contributions and then redesigns multi-head attention to optimize the credit assignment in value function factorization. To evaluate the effectiveness of our method, we conduct a series of cooperative multi-agent tasks on the StarCraft II platform and compare the results with several state-of-the-art techniques, including Qatten, QMIX, QTRAN, COMA, and VDN. The experimental results demonstrate that our method achieves faster overall convergence speed, higher stability, and robust performance across various scenarios.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.