Hyper-Realistic 3D Face Models Can Now Be Created from a Single Photograph

Deep Learning –  

Researchers have used a deep convolutional neural network system to produce photorealistic 3D renders of faces, with fine details like pores, stubble and other features, all from a single input image. The system even worked on low resolution photographs.

For centuries, sculptors have meticulously recreated faces in marble and bronze with painstaking precision, and hours and hours of labor. It is part of the mystique of art that such skill can be attained, and is part of the reason we still marvel at the artwork of ancient Greece.

In today's world artists and ateliers are just as likely to be working in the digital realm to create lifelike representations of faces and anatomy. The work can also involve a combination of the physical world and the digital, with the introduction of 3D scanning.

"We can successfully digitize historic figures that are no longer available for scanning and produce high-fidelity facial texture maps with mesoscopic skin details."

Now, new deep learning research promises to make the creation of realistic 3D models even simpler, by generating them directly from 2D photographs. Along with art, the technology has applications in entertainment, gaming, police sciences and more. For example imagine in the not-to-distant future making the character in your video game actually have your face, just by using your social media avatar to generate it.

A team of researchers from the University of Southern California and the USC Institute for Creative Technologies have developed an inference framework based on deep neural networks for synthesizing photorealistic facial textures along with 3D geometry from a single unconstrained photograph.

"We can successfully digitize historic figures that are no longer available for scanning and produce high-fidelity facial texture maps with mesoscopic skin details," they claim in their paper.

With virtual and augmented reality becoming the next generation platform for social interaction, there could be a huge demand for realistic 3D avatars that are easy to make. Such avatars could be pupeteered through facial performances, or used to replace one face with another. 

The researchers also had an eye to cultural heritage. "Iconic and historical personalities could be restored to life in captivating 3D digital forms from archival photographs," they wrote. 

"Can we use Computer Vision to bring back our favorite boxing legend, Muhammad Ali, and relive his greatest moments in 3D?" 

The researchers process starts with an initial estimation of shape and low-frequency albedo. Albedo is the base color input, commonly known as a diffuse map. An albedo map of a face defines the color of diffused light over the skin of the subject.

Their system then computed a high-frequency partial texture map, without the shading component, of the visible face area. To extract the fine appearance details from this incomplete input, the researchers introduced multi-scale detail analysis technique based on midlayer feature correlations extracted from a deep convolutional neural network. The deeper the neural network, the finer the detail that was brought out in the final render.

Related articles

• Deep Learning Looks To Take Over Artist Jobs Now 
• 3D Scan the World for Robots 
• Create Your Own Deep Dream Artworks 

The complete and photorealistic texture map was then synthesized by iteratively optimizing for the reconstructed feature correlations. Using these high-resolution textures and a commercial rendering
framework, the researchers produced high-fidelity 3D renderings that are visually comparable to those obtained with much more expensive and laborious multi-view face capture systems.

"Our inference technique produces high-resolution texture maps with complex skin tones and mesoscopic-scale details (pores, stubble hair), even from very low-resolution input images," conclude the researchers.

In addition to their own extensive evaluations, the researchers also validated the realism of their results using a crowdsourced user study. Another benefit of the work is that a new dataset of 3D face models with high-fidelity texture maps based on high-resolution photographs will be publicly available to the research community.

SOURCE  Two Minute Papers

By  33rd Square	Embed