Combining Unity with machine vision to create low latency, flexible, and simple virtual realities
Rattachement africain : au, us, de, se. Niveau de preuve : code pays fourni par la source.
Le résumé fourni par la source
Abstract In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviors. By using the actions of a constrained animal to control the visual scenery, the animal is provided the perception of moving through a simulated environment. As the animal is constrained in space, this allows detailed behavioral quantification. Additionally, as the world is generally computer-generated this allows for mechanistic quantification of visual triggers of behavior. We created a novel virtual arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi-modal virtual reality arena, MultiMoVR (Kaushik et al., 2020) but tracked hoverfly wing movements using DeepLabCut-live (DLC-live, Kane et al., 2020). For trackball experiments, we recorded the motion of a ball that a tethered crab was walking on using FicTrac (Moore et al., 2014). In both cases, real-time tracking was interfaced with Unity to control the movement of the tethered animals’ avatars in the virtual world. We developed a user-friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. We show that both the DLC-live-Unity and the FicTrac-Unity configurations close the feedback loop effectively with small delays, less than 50 ms. Our FicTrac-Unity integration highlighted the importance of closed-loop feedback by reducing behavioral artifacts exhibited by the crabs in open-loop scenarios. We show that Eristalis tenax hoverflies, using the DLC-live-Unity integration, navigate towards flowers. The effectiveness of our CAVE interface is shown by implementing experimental sequencing control based on avatar proximity to virtual structures. Our results show that combining Unity with machine vision tools such as DLC-live and FicTrac provides an easy and flexible virtual reality (VR) environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design and implement new experiments without programming in code. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.
Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.
Le contrôle bibliographique ouvert
DOI retrouvé dans Crossref DOI retrouvé ; titre concordant.
- Titre Crossref
- Combining Unity with machine vision to create low latency, flexible, and simple virtual realities
- Date Crossref
- 06/02/2024
- Éditeur
- openRxiv
- Type
- posted-content
Ce recoupement confirme des métadonnées liées au DOI. Il ne confirme ni la méthode ni les conclusions de l’étude, et il ne compte pas comme une seconde source scientifique indépendante.
Où se fait cette recherche
-
Flinders University Flinders Health and Medical Research Institute pays non établi dans la noticeUniversité ou école supérieure
-
The University of Western Australia pays non établi dans la noticeUniversité ou école supérieure
-
Ocean Institute pays non établi dans la noticeOrganisation à but non lucratif
-
Oceans Institute pays non établi dans la noticeOrganisme public
-
University of Konstanz Centre for the Advanced Study of Collective Behaviour pays non établi dans la noticeUniversité ou école supérieure
-
Max Planck Institute of Animal Behavior Department of Collective Behaviour pays non établi dans la noticeStructure de recherche
-
Uppsala University Department of Medical Cell Biology pays non établi dans la noticeUniversité ou école supérieure
-
College of Science and Engineering pays non établi dans la noticeUniversité ou école supérieure
Flinders Health and Medical Research Institute — Flinders University, The University of Western Australia et Ocean Institute, avec 5 autres affiliations.
Une affiliation ne permet pas de déduire la nationalité d’un auteur.