EGENEFACE: HIGH QUALITY GENERALIZED TALKING FACE SYNTHESIZING ENABLED BY AUDIO

Le résumé fourni par la source

A critical issue in virtual reality and filmmaking is the generation of photorealistic video portraits from random spoken recordings.The use of neural radiance fields is to enhance 3D realism and image authenticity has been the subject of several recent studies.Nevertheless, the limited training sample scale of earlier NeRF-based techniques restricts their applicability to out-ofdomain sounds.The previous model builds upon GeneFace, which introduced variational auto encoder-based framework with Neural Radiance Field (NeRF) rendering for lip-syncing tasks.While GeneFace demonstrated strong results in terms of synchronization accuracy and visual quality, several limitations hindered its performance in challenging scenarios such as noisy audio, out-of-domain speech, and temporal in stability.Our proposed model, EGeneface, addresses these gaps though.In terms of Generator of variational motion, we reduced Landmark Distance by 5.9% and improved lip synchronization accuracy.For advanced renderer for NeRF, we lowered Frechet Inception Distance to 20.15, improving the visible realism of created videos.In training with noise to increase robustness, we increased Sync Confidence in OOD scenarios by 12%, achieving more reliable lip-syncing across diverse audio environments.For Real-Time deployment optimization, we improved frame processing rate from 30 FPS to 35 FPS, making suitable for real-time applications.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.