LLM driven AI avatars : a framework for emotionally adaptive interaction in virtual reality

Abstract

Creating emotionally responsive AI in virtual reality (VR) remains a challenge, as modeling or replicating human emotional behavior is inherently complex. This thesis explores how large language model (LLM)-driven avatars, personalized using self-reported traits, can simulate emotionally adaptive interaction in immersive environments. A complete system was developed that generates custom 3D avatars with real-time lip synchronization, based on facial scans and personality data provided by users.

Five participants contributed personal information to guide the avatar’s dialogue style through prompt engineering. Each interacted with their digital clone in VR, and their experiences were evaluated through structured questionnaires and open-ended feedback. The results revealed two key findings. First, immersion and embodiment significantly improved the feeling of emotional presence. Second, participants strongly prioritized the avatar’s ability to act like them in terms of language and tone. Interestingly, a gap was observed between the way participants described themselves and how they were perceived by the researcher, who already knew them. This raises questions about the accuracy of self-reported traits in AI modeling and whether third-party input could help improve personality alignment.

These findings contribute to the design of emotionally aware AI agents for applications in therapy, education, and virtual communication. The study also emphasizes the importance of identity, consent, and perception when creating AI representations of real people.