Facebook AI Research team unveiled an AI called Vid2Game that grabs real people from videos and turns them into playable game characters, similar to those ’80s full-motion video games (Night Trap, Ground Zero Texas, Prize Fighter, Corpse Killer, etc.). Put simply, it uses neural networks to analyze random videos of people performing specific actions and then proceeds to recreate that character and action in virtually any environment. Read more for a tech demo video and additional information.
“The team used two neural networks called Pose2Pose and Pose2Frame. First, a video is fed into a Pose2Pose neural network designed for specific types of actions like dancing, tennis or fencing. The system then figures out where the person is compared to the background, and isolates them and their poses. Then, Pose2Frame takes the person, along with their shadow and any objects they’re holding, and inserts them into a new scene with minimal artifacts. You can then control their movement, based on poses from the video, using a joystick or keyboard,” according to Engadget.