Unmanned Swarm System Decision-Making Based on Heterogeneous Mean Field Reinforcement Learning

Résumé fourni par la source

Finding the optimal combat strategy is a challenging problem in unmanned swarm system. For sequential decision-making tasks, Multi-Agent Reinforcement Learning (MARL) offers an effective solution and a promising framework for developing intelligent responses. However, applying MARL to large-scale swarm confrontations presents three critical challenges: i) the curse of dimensionality caused by the excessive scale of the swarm, ii) the generalization difficulties caused by the dynamic changes in swarm size, and iii) the limitations in policy sharing due to the heterogeneity of agents in terms of combat capabilities and mission objectives. To address these challenges, we propose a novel MARL paradigm called Heterogeneous Mean-Field Reinforcement Learning (HMFRL). In this approach, the bidirectional interactions between any agent and its neighboring agents are modeled as interactions between a central agent and a virtual agent, which abstracts the average effect of its neighbors. This method simplifies the multi-agent problem into a two-agent problem, which can reduce the input dimensionality of the agent and adapt to the changing swarm size. Furthermore, dedicated policy networks are developed for different types of heterogeneous agents, along with task-specific state spaces, action spaces, and reward mechanisms. A team reward mechanism is also introduced to enhance coordination among heterogeneous agents. By integrating this paradigm with Double Q-Learning and Actor-Critic algorithms, we propose heterogeneous mean field Q-learning (HMFQ) and heterogeneous mean field Actor-Critic (HMFAC) algorithms. Experimental results in a swarm confrontation environment demonstrate the effectiveness and scalability of our algorithms.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.