Characterizing the Effectiveness of DINOv2 as an Off-the-Shelf Foundation Model for Earth Monitoring Tasks: Preliminary Results

Le résumé fourni par la source

Earth monitoring has important applications across major social sectors including urban planning, agriculture, ecology, etc. Many of these applications rely on semantic segmentation, where the goal is to generate pixel-level classifications (e.g., buildings and crops) to inform decision-making. Developing a general-purpose model for the problem is often challenging at large scale, given the often limited ground truth labels (e.g., many tasks require field surveys), cross-region variability, and different sensor characteristics. Recent progress in industrial general-purpose vision foundation models (VFMs) has shown encouraging potential on a wide range of downstream tasks in the natural image domain. However, existing evaluations have not characterized the effectiveness of these off-the-shelf industrial VFMs on remote sensing segmentation tasks to understand the status and suitability for different types of Earth monitoring tasks. This is particularly important considering that the end goal of the general-purpose VFMs is to reach zero-shot generalizability to broad tasks like in the language domain. In this work, we present preliminary results on the performance of DINOv2, a state-of-the-art industrial VFM, for remote sensing image segmentation. We hypothesize that off-the-shelf DINOv2 performs well on tasks where targets have clear texture patterns and sharp boundaries, such as building footprint mapping, but may struggle with targets having fuzzy boundaries or a lack of texture signature. We consider DINOv2 to be effective if it can achieve comparable or better results compared to a regular-sized supervised U-Net model, and ineffective if there exists a large gap. The preliminary results align with our expectations, highlighting both the potential and current limitations of industrial VFMs in remote sensing segmentation tasks when there are limited textures and the boundaries are softer.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.