Google DeepMind’s SIMA 2 Steps Into the Game, Virtual 3D Worlds That Is

Google DeepMind has just unveiled SIMA 2, an AI agent that dives headfirst into 3D video games and handles them like a pro in no time. Last year’s SIMA 1 could handle some basic directions in virtual settings, but until recently, it was only succeeding about 31% of the time when things got tricky, whereas humans were able to crack it at a rate of 71%. With SIMA 2, they’ve approximately quadrupled that success rate by combining it with Gemini’s language intelligence, which essentially allows it to take raw commands and transform them into a meticulously plan, along with all the back and forth communication.

The researchers fed it a ton of video footage of people playing games from partners like Coffee Stain Studios – think Valheim, Satisfactory, Goat Simulator 3 or Hello Games’ No Mans Sky, along with a few others like Space Engineers & Teardown. Then, SIMA 2 just watches the screen, picks up the virtual keyboard and mouse and sets to work, all without having to stare at any of the game code. Tell it to climb a ladder or pull up a map, and it nails this sort of basic task over 600 times. And with Gemini’s help, it’s able to break down bigger goals – like finding a campfire in the Viking survival game ASKA – even though its never seen that place before.

Joe Marino, a senior research scientist on the team, ran a demo on No Man’s Sky live in front of everyone. SIMA 2 checked out a rocky planet, spotted a distress signal and laid out its plan of attack for reaching it. Then, another clip showed how when asked to head to the house with the same red as a ripe tomato, SIMA 2 took a moment, figured out the fact that tomatoes are red, and marched itself straight to the red one . Emojis work too – so you can send 🪓🌲 and it’ll just go swing that axe at a tree.


You can actually chat with SIMA 2 through text, voice, or even just drawing little pictures on the screen. It kind of explains what its next move will be, asks for clarification if it needs any, and honestly just feels like a player you’re teaming up with, rather than just some robot following orders. Jane Wang, another team member with a background in neuroscience, pointed out how tricky it is to work out the flow of the game, interpret what you’re saying, and still manage to react sensibly every time – it’s no easy feat.


But what SIMA 2 really shows off is its ability to excel when its thrown into new territory. Put it into MineDojo, a Minecraft copycat it never even saw during training, and it just crushes the tasks that SIMA 1 would stumble on. The results show it closing the gap with human scores across the taught games, and actually starting to make some real gains even on holdouts like ASKA. And when paired up with Genie 3, DeepMind’s tool for whipping up 3D environments from text or images, SIMA 2 can just take its bearings in a complete strangers yard and follow orders without batting an eye.


Self growth is what sets SIMA 2 apart and pulls you deeper into its world. Start with human gameplay clips as a base, then let Gemini create missions and score attempts. The agent attempts, fails, logs the run and then retrains using its own collection of experiences. Repeat and it masters what previously stumped it – no new human clips required. Marino called it a big step towards robots and general intelligence. Even in Genie spawned worlds it continues to hone skills with endless loops of play and feedback.

Games are the perfect sandbox for these skills – navigation, tool use and teamwork on goals – all of which translate directly to real robots. Frederic Besse, senior staff engineer, explained it: send a home bot to count bean cans in the cupboard and it must know beans, recognize the cupboard and navigate the path.