A Reinforcement Learning Approach for Long-term Hydrothermal Dispatch

Résumé fourni par la source

Abstract Hydrothermal power-system management involves high-dimensional uncertainty (curse of dimensionality), challenging traditional stochastic programming. Stochastic dynamic programming (SDP) and scalable variants remain robust industrial baselines but face rising computational strain as grids add flexibility and variability. This paper presents an enhanced reinforcement learning (RL) framework for long-horizon hydrothermal dispatch, extending tabular approaches via a scalable Gymnasium-based representation of the Uruguayan power system. Alongside a tabular Q-learning baseline, it introduces a proximal policy optimization (PPO) agent for continuous state-action control, representing reservoir volumes and turbine releases without discretization. Optuna-based hyperparameter optimization is applied to both methods, and stochastic results use multiseed ensemble statistics. Performance is evaluated under deterministic inflows and under stochastic settings using a nonhomogeneous Markov inflow model (in-sample) and century-scale historical chronicles (out-of-sample). Within the adopted environment, PPO achieves a 3.96% lower cost than Q-learning (2328.17 vs 2424.28 MUSD), and both are close to the MOP reference (2463.64 MUSD). Out-of-sample historically, Q-learning is within 1.29% of MOP (2761.25 vs 2726.07 MUSD), while PPO attains 2645.01 MUSD within the adopted simplified environment. The purpose of this study is methodological rather than operational: it does not aim to provide a production-ready solution to the full hydrothermal scheduling problem, but to assess whether RL agents can learn stable, robust, and physically meaningful long-horizon dispatch policies in a controlled simplified environment. Comparisons must consider the model gap: MOP includes head-dependent hydropower physics, whereas the RL environment assumes a constant energetic conversion coefficient, affecting low-head costs. Accordingly, the reported benchmarks should be interpreted as evidence of convergence and policy coherence under transparent simplifying assumptions, rather than as a claim of full operational equivalence or deployment readiness.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.