I built marshmallow castles in Google’s new AI world generator

Google DeepMind is opening up access to Project Genie, its AI tool for creating interactive games from text or images.

Starting Thursday, Google AI Ultra subscribers in the US can play around with the experimental research, which is powered by Google’s latest global integration. Gene 3its photo maker Nano Banana Pro, and Gemini.

Coming five months after Genie 3’s exploratory review, the move is part of a broader push to gather user feedback and academic knowledge as DeepMind competes to create more reliable global models.

Global models are AI systems that create an internal representation of the environment, and can be used to predict the future and plan actions. Many AI leaders, including those at DeepMind, believe that global models are a critical step towards achieving artificial intelligence (AGI). But recently, labs like DeepMind are seeing a system going to market that starts with video games and other entertainment and branches that are trained by virtual assistants (aka robots).

DeepMind’s release of Project Genie comes as global competition heats up. Fei-Fei Li’s World Labs late last year released its own The first product called Marble. Runway, an AI video startup, is also here launched a national model recently. And the former chief scientist of Meta Yann LeCun founder of AMI Labs it will also focus on developing international models.

“I think it’s great to be in a place where we can have a lot of people to find us and give us answers,” Shlomi Fruchter, director of research at DeepMind, told TechCrunch via video interview, smiling from ear to ear with the joy evident at the release of Project Genie.

The DeepMind researchers that TechCrunch spoke to were upfront about the tool’s testing. They can be inconsistent, sometimes producing impressive performances, other times producing dull results that miss the mark. This is how it works.

You start with “world art” by providing information about the environment and the main character, who you can later control around the world in first or third person. Nano Banana Pro creates an image based on what you can edit, intuitively, before Genie uses the image as a jumping off point into the interactive world. The changes usually worked, but the model would sometimes stumble and give you purple hair when you asked for green.

You can also use real life images as the basis of a world building model, which is hit or miss. (More on that later.)

Once you’re satisfied with the image, it takes a few seconds for Project Genie to create a familiar world. You can also mix existing worlds for new releases by building on top of what they tell you, or search for worlds in the archives or by using the randomizer tool boost. Then you can download videos of the country you are looking for.

DeepMind is only offering 60 seconds of global generation and navigation at the moment, partly due to budget and computational constraints. Because Genie 3 is auto-regressive modelit takes a lot of dedication – which puts a hard ceiling on what DeepMind can deliver to users.

“The reason we cut it to 60 seconds is because we want to bring it to more users,” Fruchter said. “Basically when you use it, there’s a chip somewhere that’s just yours and it’s assigned to your part.”

He also said that extending it beyond 60 seconds would reduce the quality of the test.

These places are impressive, but sometimes, due to their interactions and the forces of nature, they are limited.

Stupidity works, reality doesn’t

When I used the example, the security features were already working. I couldn’t make anything like a nudity, nor could I make worlds that smell like Disney or other copyrighted material. (In December, Disney hit Google with a quit-and-quitchallenging the company’s AI models of copyright infringement by teaching Disney characters and IP and making illegal content, among others.) I couldn’t even get Genie to create a world of mermaids exploring fantasy underwater worlds or ice queens in their winter homes.

However, the show was very impressive. The first world I built was an attempt to have a little childhood imagination, where I could explore a castle in the clouds made of marshmallows and a river of chocolate sauce and trees made of candy. (Yes, I was a little kid.) I asked the model to do it in the style of claymation, and it gave a world of knowledge that my childhood would have eaten, the pastel-and-white spiers and turrets looked puffy and delicious enough to tear off a chunk and put it in a jar of chocolate. (Video above.)

That said, Project Genie still has some challenges to overcome.

These models excelled in creating worlds based on artistic techniques, such as the use of watercolors, anime style or classic cartoon textures. But it tends to fail when it comes to photorealistic or cinematic worlds, often coming out looking like a movie set instead of real people in real life.

And it is not always a good response when given real pictures of work. When I gave it a picture of my office and asked it to create a world using the picture exactly as it was, it gave me a world with the same furniture of my office – a wooden desk, plants, a gray sofa – laid out differently. And it seemed sterile, digital, not alive.

When I fed it a picture of my desk with a stuffed toy, Project Genie showed the toy as it navigated through space, and sometimes other objects would react as it passed by.

That connection is something that DeepMind is working to improve. There were a few times where my characters went through walls or other solid objects.

When DeepMind first released Genie 3, researchers highlighted how autonomous designs meant they could remember what they had created, so I wanted to test that by going back to areas they had already created to see if they would match. For the most part, the model succeeded. One time, I made a cat while checking another desk, and only once I went back to the right side of the desk did the color make a second cup.

The part I found most frustrating was how you navigate the area using the arrow keys to look around, the airspace to jump or climb, and the WASD keys to move. I’m not a gamer, so this didn’t come naturally to me, but the keys often didn’t respond, or sent you in the wrong direction. Trying to walk from one side of the room to the door on the other side is often a mess, like trying to steer a cart with a broken wheel.

Fruchter assured me that his team is aware of these weaknesses, and reminded me that Project Genie is an experimental model. In the future, he said, the team hopes to improve reality and improve communication skills, including giving users more control over events and locations.

“We don’t think of (Project Genie) as the ultimate drug that people can come back to every day, but we think there is already a vision of something exciting and unique and it couldn’t be done any other way,” he said.