New Wizardry Allows Researchers To Turn Photos Into Three-Dimensional Objects

Researchers at Carnegie Mellon University have created a way to manipulate objects in photos in three dimensions, allowing you to see all sides of formerly 2D objects. How is it done? Some might say there is dark magic afoot, but what’s really happening is far more interesting.

Here’s how it’s done: an object is selected in an image, be it a chair, an origami crane, or a fireplug. The system matches the object with currently extant 3D models taken from various sources, and then, by connecting the models with the actual objects, they are able to simulate what the object would look like in the photograph. While this database of objects is obviously fairly limited, it does allow for some clever tricks including making taxi cabs in photos flip around to display their undercarriage and then zoom off into space.

You can think of it as a sort of 3D modeling shorthand. While it’s impossible for the computer to interpolate sides of an object it cannot see, it can easily match a ready-made 3D object with the object in the picture. Weird objects like hands and faces might be tough (unless you’re me) but common things like cans, bottles, and furnititure might already exist as a 3D model. Using a bit of texture mapping and lighting tricks, you can easily replace a MacBook in a photo with a 3D-rendered one.

”This three-dimensional manipulation of objects in a single, two-dimensional photograph is possible because 3-D numerical models of many everyday objects — furniture, cookware, automobiles, clothes, appliances — are readily available online. The research team led by Yaser Sheikh, associate research professor of robotics, found they could create realistic edits by fitting these models into the geometry of the photo and then applying colors, textures and lighting consistent with the photo.

“Instead of simply editing ‘what we see’ in the photograph, our goal is to manipulate ‘what we know’ about the scene behind the photograph,” said Natasha Kholgade, a Ph.D. student in the CMU Robotics Institute. The researchers find that most basic objects work fine in the system but often objects are worn down, discolored, or unique. To fix this, they grab the visible shell of the object in one position, interpolate what it will look like in three dimensions, and then add similar details based on the models. The key, then, is to be able to find and alter the 3D models quickly enough to ensure that the objects are properly positioned in the photograph as needed, a tall order for now. The next step, said the researchers, is to create a 3D object searching algorithm that will make it easier to find objects on the fly.