The global version of Google Genie can now simulate real streets with Street View

We’ve all pulled up Street View on Google Maps to show a friend what our childhood home looked like, or we’ve dropped a photo of the little guy on the streets of Paris to see if we’ve rented a hotel in the right place. Imagine being able to do that, but more seriously, an interactive system that allows you to really test the road and the surrounding area, and do things like change the weather or see how it would look in the “Day After Tomorrow” scenario.

This is one of the goals of Google’s latest integration. Starting today, Google DeepMind is connecting Street View with Project Genie, a global global brand of companies that can create a diverse, interconnected environment. The new features were introduced at the Google I/O developer conference.

“It’s very powerful for the user (and robotics) and for people to play with, and it’s always been a Genie idea,” Jack Parker-Holder, a research scientist at the open source group DeepMind, told TechCrunch.

He gave an example of a new robot being deployed in London, which often sees the sun. Genie can, Parker-Holder says, capture the rare moments when the sun shines on Victoria’s home, so the glare doesn’t startle the robot when it happens.

“At the same time, you can say, ‘I’m going to New York City, but not this time of year,'” he continued. “‘Being in the snow, I want to see how the monument looks in the snow.’

Google has been collecting Street View data for 20 years through cars equipped with cameras and people wearing them with “tracker bags.” The technology giant has collected north of 280 billion images in 110 countries and seven continents.

“With Street View, we have images from many countries,” Jack said. “You can imagine how powerful it is to combine a rich source of real world information and data and be able to model countries.”

Google has released its latest version to the world Genie 3 to preview the survey August is over and they opened access to the tool for Google AI Ultra subscribers in the US in January, allowing customers to create interactive game worlds from audio or images. The goal is to use Genie in education, gaming, and robotics education.

Genie 3 is already helping power one of Waymo’s simulators training self-driving cars for “extremely unusual situations” such as hurricanes or encounters with wild elephants. Putting Street View data there could help Waymo plan to roll out in more cities around the world.

Waymo has its own prototype that it relied on to reach 11 US cities and test its AI driver in several others. The difference with the Genie, says Parker-Holder, is that they both come from a car environment. Street View allows not only to compare the fixed world to a real place, but also to transfer the view to other types of agents, such as a person or a robot.

Google is launching Street View in Genie for some Ultra users in the United States starting today, with long-term access. Global Ultra users will get access over the next few years, according to the company.

The goal of the researchers is to put this new technology into as many hands as possible, according to Diego Rivas, a product manager at DeepMind. He cautioned that Street View and especially Genie are still experimental, so there is much to be done to make it more accurate.

In the examples that the Google team showed me – including simulations in the waters of the area where I lived – the results are impressive and familiar, but the quality of video games and not photorealistic. These models also do not know physics, meaning they do not understand cause and effect. For example, when simulating a woman running through a snowy Joshua Tree tree, she ran through cacti and bushes.

Compare this to, say, Google’s Nano Banana image generator – which can now create great text in infographics – or its video generator Veo – which understands that paper boats float on water waves, smoke disperses in the air, and fabrics drag on colors.

Physics is not hard-coded in these examples; they learn intuitively over time through mere observation, just as a living person would.

“I think for this type of thing, it’s probably six to 12 months behind the video in terms of accuracy and quality, so I think that’s something we’re going to deal with,” Parker-Holder said.

Jonathan Herbert, director of Google Maps who started on the Street View team as a student 12 years ago, said Genie still can’t faithfully reconstruct a street. He thinks the real breakthrough is the advancement of space for AI. When you turn 360 degrees, the AI ​​correctly remembers and simulates the environment behind you. From there, the model can build a new site on top of it.

“We thought long and hard about how we could build the world’s best and richest model on top of Street View data,” Herbert said. “It has been our idea to use Maps Data in new ways and for new types of AI research for a long time.”