This neural network can make a 3D face from a single photograph

Thomas McMullan Read more December 8, 2016

Researchers at Cornell University have come up with a way to make somewhat unsettling 3D renders of faces, using only a single photograph.

While facial mapping normally requires a series of images, each in ideal lighting conditions, the researchers instead used an extensive database of other people’s faces in co-ordination with a neural network.

In their own words, they showed that “fitting a convex combination of feature correlations from a high-resolution face database can yield a semantically plausible facial detail description of the entire face”. Parklife.

In a nutshell, the neural network was able to make the render by pulling on the facial database for comparable textures, then blending these together and layering them on top of its estimation of a face shape. As you can see from the video below, they fed in pictures of famous figures – including Donald Trump, Hillary Clinton, Muhammad Ali and Joy Division’s Ian Curtis, as you do – and the system turned these into eyeless 3D masks. 3d face trump.gif

The researchers say this method could one day be used to make 3D avatars in virtual and augmented reality without the need for full body scanning. They also mention the possibility of virtually resurrecting historical personalities from archive photos. “Can we use Computer Vision to bring back our favorite boxing legend, Muhammad Ali, and relive his greatest moments in 3D?” they ask.

Facial-scanning technology – such as Intel’s RealSense camera – is already enabling real-time facial scanning and puppeteering, such as in the RSC’s current production of The Tempest. The system put forward by the researchers from Cornell, however, suggests that realistic digital avatars can be made without the need of advanced sensors.

Alongside this is a heap of questions about privacy – not least the appropriation of other people’s faces in virtual worlds. If I can inhabit a realistic replica of a person’s body using little more than their Facebook profile picture, what’s to stop me from masquerading as them, or harassing other people while I look like them?

Researchers at Carnegie Mellon University recently showed how specially designed glasses can dupe facial-recognition systems. While the Cornell University project raises questions about identity in virtual reality, both suggest that appearances may be deceiving in a world of facial sensors and digital avatars.