AI that makes 3D models from 1 2D photo?

In order to convert sketches and photographs of objects into realistic 3D models and output them to 3D printers, make them move in TV games, or use visual effects to bring video to life, you need talented people with digital modeling skills who work on mass images. Do.

However, Nvidia has trained a neural network to create a perfectly textured 3D model from a single photo. Of course, there was a similar technology that automatically generated 3D models before. However, to get accurate results, many pictures taken from different angles were required, and humans needed to enter numbers to understand the specific object size and shape of the software image.

You can approach problem solving in any way, but it’s welcome to be able to do 3D modeling more simply. It can be a tool that can be used by a wide range of users even without advanced skills.

In a new paper (Learning to Predict 3D Objects with an Interpolation-Based Renderer) published by NVIDIA’s research team during the annual Neuroinformation Processing System conference in British Columbia, Canada, a new graphic tool DIB-R (Differential interpolation-based renderer) was developed. It explains in detail.

The team trained a DIB-R neural network with several datasets. This includes photos previously converted to 3D models, 3D models presented from different angles, and sets of photos focused on specific subjects from different angles. For example, it takes about two days for a neural network to train how to convert a specific subject, such as a bird, from 2D to 3D. However, once completed, it is said that 3D models can be mass-produced within 100 milliseconds based on 2D photos that have not been analyzed so far.

This amazing processing speed makes this tool more interesting. This is because there is a possibility that the method of recognizing that something like a robot or an autonomous vehicle is in front of us and understanding it in 3D will be greatly improved. For example, a still image created from an image taken with a camera by an autonomous vehicle is directly converted into a 3D model to accurately measure the size of a large truck that needs to be solved from now on, and how the robot will appropriately select an arbitrary object according to the planned shape. Become predictable.

DIB-R can also improve the performance of surveillance cameras that identify and track people in order to facilitate the image data of people moving through the scene, with 3D models generated instantaneously. Related information can be found here .