Weather Data Streaming with Kerchunk: Strengthening Early Warning Systems

Le résumé fourni par la source

The Ensemble Prediction System (EPS) provided by global weather forecast centres generates vast amounts of data that is crucial for early warnings of extreme weather and climate. However, regional and national meteorological services often face challenges in processing this data efficiently, particularly during regional downscaling and post-processing. Conventional methods of downloading and storing GRIB-format data have become increasingly inefficient and unsustainable. The Strengthening Early Warning Systems for Anticipatory Actions (SEWAA) project aims to address these challenges by exploring the use of cloud native operations and GenAI-cGAN driven post-processing systems. Kerchunk provides a groundbreaking solution for real-time weather data streaming, catering to the transition towards open and free to use cloud-based object storage from global weather forecasting centres. Kerchunk, in conjunction with GRIB index files, enables efficient, real-time access to weather data, fostering more sustainable workflows in weather and climate services, thus strengthening early warning systems. This study developed a workflow for streaming forecast data using Kerchunk with two primary objectives: 1. Using GRIB index files to reduce redundant readings and generate Kerchunk reference files. 2. Through streaming-like access, convert the reference files into virtual Zarr datasets and utilise Dask compute for scalable data handling The methodology utilised recent improvements in the Kerchunk library that integrate GRIB scanning with its index files. This allowed the system to sample subsets of the GRIB corpus instead of processing entire Forecast Model Run Collections (FMRC), significantly optimising performance. The workflow was implemented using cloud-based compute operations via Coiled python library and its service on the Google Cloud Platform. Dask cluster, managed through Coiled, enabled the creation of Zarr virtual datasets for analysis and visualisation. This streaming approach efficiently loads weather data into memory on demand, avoiding unnecessary data downloads and duplication. We validated the solution with NOAA GFS/GEFS datasets stored in AWS S3 bucket as open datasets. The optimised workflow demonstrated remarkable efficiency, requiring only

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.