A Human-in-the-Loop Approach to Learning Social Norms and Behavioural Policies

Le résumé fourni par la source

Abstract It is often desirable to constrain reinforcement learning (RL) policies to align with societal norms and individual preferences in order to better represent users’ intentions and expectations. In order to adequately deal with exceptions and conflicts between competing norms/preferences, it is useful for such constraints to be defeasible; and to enable transparent justification for decisions, as well as allowing them to be more easily re-used in different variations of a particular problem setting, it is also helpful to express them in a high-level, symbolic, human-understandable form. However, manually specifying these norms can be challenging, typically requiring familiarity with a logical formalism and extensive knowledge of the application domain. To address this, we propose a human-in-the-loop approach that learns ethical constraints by combining autonomous exploration with expert imitation from simple user-provided examples of the intended behaviour. Our system infers a list of high-level logical constraints along with an RL policy that adheres to them, while optionally allowing users to interactively refine constraints, resolve conflicts, manage ethical contradictions and provide counter-examples. The system interleaves an Approximate Q-learning (AQL) component for goal-directed exploration and an interactive Inductive Logic Programming (ILP) module for symbolic constraint inference, thus ensuring learned norms remain transparent, auditable, and editable. We evaluate our method in a Pacman environment, demonstrating that the learned logical constraints achieve normative compliance comparable to existing approaches, without requiring manual rule specification. Furthermore, we demonstrate the value of human intervention via interactive ILP mechanisms, and experimentally show that it accelerates convergence to accurate and efficient normative frameworks. This work therefore contributes a novel approach to learning symbolic, defeasible constraints for RL policies, and introduces interactive mechanisms that allow constraints to be tailored, refined, and overridden in accordance with user intentions.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.