While Lucasfilm hired the VFX artist who created Star Wars deepfake videos on YouTube, NVIDIA’s neural talking-head synthesis took a different approach. This model takes deepfakes to the next level as it’s capable of synthesizing a talking-head video using a source image containing the target person’s appearance and a driving video that dictates the motion in the output.
This motion is then encoded based on a novel key point representation, where the identity-specific and motion-related information is decomposed without supervision. Extensive experimental validation showed that their model outperforms competing methods on benchmark datasets. Plus, their compact key point representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth.
Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating a face-to-face video conferencing experience,” said the researchers.
