A new patent filed by Meta showcases technology that could lead to more realistic lip-syncing and facial animation for video games and VR applications. The creation of NPCs and player avatars with these more lifelike facial movements opens the door to more realistic, engaging, and immersive games and experiences, including inVR games.
Even as graphics have advanced in video games over the decades, allowing the creation of nearly photorealistic characters in recent years, portraying realistic speech and facial expressions for those characters has continued to provide a challenge to developers. Gamers' brains are accustomed to seeing and reading the faces of other people, so even the best facial animation on a video game character can easily cross intouncanny valleyterritory, leading to characters looking “off” or even downright disturbing. Meta’s patent could allow for the generation of realistic lip-syncing and facial expressions on the fly, saving time for developers and increasing immersion for players.
RELATED:Facebook Working on System to Allow Users To Scan Their Clothes into the Metaverse
The patent details a variety of methods for translating a user’s speech into realistic lip-syncing and facial animation, giving developers flexibility in how they choose the use the system. One method outlined in the patent involves using a dataset of audio and video recordings of multiple people as they read 50 different phonetically balanced sentences, with the system tracking their facial expressions as they read through each. By tracking how individuals’ faces and mouths move as they read through each sentence, the system can then pull from these movements and blend them together to realistically animate a character, even including subtle movements like blinks and eyebrow raises.
With upcoming VR headsets such as theMeta Quest Proand Pico Neo 4 Pro including built-in face and eye tracking, it’s one of the other possibilities described in the patent that fans of VR games and social experiences might find the most interesting. In addition to being able to generate facial animation and lip-syncing based on an existing dataset, the patent also describes how the system could do the same thing using cameras integrated into a VR headset. This feature could allow VR users to have avatars that accurately mimic their own facial expressions, making for a more realistic and immersive experience in multiplayer games and social VR experiences like VR Chat.
From Valve’s Source engine, which used an animation system that modeled facial movements for each phoneme (the individual sounds that make up words) of dialogue, to Team Bondi’s use of MotionScan technology to captureL.A. Noire’s actors’ performances using a system of 32 high-definition cameras, creating realistic faces for gaming has always seemed just out of reach. Meta’s patent could indicate a huge technological leap forward in this quest, and may help usher in a new era of high-quality, lifelike facial animation in games and VR experiences.
