From words to 3D faces : a real-time pipeline for avatar expression animation

Abstract

Generating realistic and controllable 3D facial animations from textual descriptions remains a challenging task, often requiring complex mappings from text embeddings to the latent space of 3D Morphable Models. This thesis demonstrates the possibility of integrating state-of-the-art image diffusion models and 3D face reconstruction for generating text-based animatable avatars and the practical challenges of doing so. My approach takes a neutral 2D facial image and a text prompt describing a desired expression as input. These are fed into a fine-tuned text-to-image inpainting model (based on DiffEdit with LoRA on the KDEF dataset) to generate a 2D image of the face exhibiting the target expression. To ensure texture consistency, a skin tone matching technique is applied to the generated image with respect to the original neutral face. Subsequently, both the original and the generated 2D images are processed using a 3D reconstruction network (DECA) to obtain textured 3D meshes. For enhancing visual realism, a post-processing phase refines the reconstructed 3D model features. Finally, continuous animation is achieved by interpolating among the 3D meshes that represent different textual input, enabling real-time text-based control over 3D facial expressions, demonstrated in a Unity context. This hybrid 2D-3D approach presents a practical and potentially more natural method for generating dynamic 3D facial animation from text input, with potential application to a range of applications in virtual reality, character animation, and human-computer interaction.