Adapting SAM3 for 3D fruit counting with cross-view contrastive learning and Hough voting

Le résumé fourni par la source

Fruit detection and 3D localization have been shown to be critical for precision agriculture applications, such as yield estimation and robot harvesting tasks. However, real-world agricultural settings can be associated with high density of fruits, strong occlusions, and variable illumination, leading to difficulty in accurately detecting and localizing fruits. Although there exist 2D approaches dedicated for fruit counting, these approaches may easily merge adjacent fruits due to projection effects in dense occlusions, thus limiting their effectiveness. Motivated by this, in this paper, we explore the problem of 3D fruit detection and localization with vision foundation model adaptation and 3D Gaussian-based scene representation. Particularly, we use low-rank adaptation to leverage SAM3 to generate high-quality instance masks with different crops and illuminations. Meanwhile, to address noisy pseudo label issue, we adopt cross-view contrastive learning with uncertainty weighting for enhanced 3D feature representation. Moreover, we devise a Hough voting procedure in the 3D Gaussian space to integrate multi-view evidence of occluded fruits, followed by clustering to estimate the number of fruits as well as their 3D locations. The proposed approach has been extensively tested with thousands of fruits from synthetic orchard scenes, real apple orchard datasets, as well as greenhouse sweet pepper scenes. Experiment results show significant performance gains for densely occluded cases, with the F1-scores reaching 0.971 and 0.977 for plum and mango, respectively. Meanwhile, our approach can achieve more than a twofold boost for 3D instance segmentation over FruitNeRF. More notably, the adapted weights and the post-processing settings transfer from the open-field apple orchard to greenhouse sweet pepper scenes, where only the SAM3 text prompt is changed and the 3D Gaussian representation is still optimized for each new scene.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.