It is no secret that NVIDIA is one of the companies that has the most views towards the future … and it has made it clear to us again. Yesterday we were able to attend the presentation of GANverse 3D, a tool developed by the NVIDIA AI Research Lab in Toronto which is a breakthrough for computer animation and video game development studios.
Did you know that you can create a three-dimensional model of an object using just a 2D image? Yes, with GANverse 3D it can be done, and in as low as 65 milliseconds (by image), although at the moment it is in the testing phase. We are going to tell you about the presentation and the great news that we could see there (we were left with the desire to see Pac-Man in 3D, but it will soon be a reality).
Richard Kerris Y Sandy FidlenNVIDIA members put the icing on the cake to their week on AI technology and 3D image recreation with this presentation, which began with a small sample of what GANverse 3D can do. At the moment it is capable of generating a multitude of three-dimensional models at the same time, after a quick upload process for a conventional image and its conversion to 3D.
”We convert a GAN model into an efficient generator data model, so that we can create 3D objects from any 2D image in the web” says Wenzheng Chang of NVIDIA. For his part, Jun Gao highlights the importance of working with real images, rather than computer-generated models: ”Because we train with real images instead of the typical pipeline, which is based on data, the AI model best generated using real world images”
An indication that was made to us from NVIDIA is that, at the moment, GANverse 3D is being used to recreate car models (something that even one of the journalists commented, about how important it could also be for the advertising campaigns of the firms), but we never imagined that we could see KITT, from The Fantastic Car, in a 3D sequence fully generated by AI and this tool.
We were shown this sequence (which you can see above), where it is explained quickly how GANverse 3D works and his great recreation of an icon from the 80s. To start, you need an ordinary 2D image of an object (In the future we can recreate what we want, but for the moment it is being applied to cars and vehicles).
Once we insert this image, the GANverse 3D program extracts a highly detailed three-dimensional image, using a procedure of ”reverse engineering‘. The process is structured in fixing the structure of the object, the textures, the movements and animations, and, of course, additional information such as the environment that surrounds it.
After generating a 3D duplicate of this image, the tool is able to set animations and movements within the ” environmental ” framework that we have set. For example, in the KITT demo we can see how desert dust kicks up when starting the car. From NVIDIA they affirm that, over time, it will be possible to generate animated models of greater realism and technical quality through AI.
To generate this 3D sequence from KITT, GANverse 3D makes use of an AI platform called Omniverse Connectors, which allows to generate environments and scenarios with ray tracing and animations in real time. The 3D image generated by GANverse is located in this three-dimensional environment created by Omniverse, making possible sequences generated entirely by NVIDIA AI technology.
The purpose of GANverse 3D is not only a giant step forward for facilitate video game development and animated sequences, but it can also help studios to remaster old games, apply technical improvements or give Ray Tracing support to many titles, as we have seen also with NVIDIA’s DLSS technology in the service GeForce Now for PC.
Speaking of more technical data, NVIDIA also explained to us how the GANverse 3D tool works. To recreate a three-dimensional model of a 2D image, a synthetic 3D data set is used, such as ShapeNet. Using photos of cars taken from the web as a reference, it is possible to use a neural network to create different layers of the model. The final model, consisting of 55,000 images, surpasses that generated by other tools, presented in Pascal3D.
In the round of questions, after the presentation and the video of KITT, some very interesting questions were thrown. For example, how copyright will affect this tool. NVIDIA officials say it is possible to change any rendering details (as trivial as it may seem), which is a great advantage to avoid problems in the legal framework.
/*–> * /
The truth is that it was a pleasure to see the legendary KITT recreated with NVIDIA AI technology, with a promising GANverse 3D tool that points out ways. In the future it is expected to be able to show other demos of moving animals, other objects and even facial expressions, although we do not know how it will affect the 65 millisecond rate or the numerical limitation to generate images.
NVIDIA shows that it remains at the forefront of the maximum gaming technology and every possible improvement in the field of video game development, and we really want to see what else it can offer GANverse 3D (there will be two new events, one in May and one in June) and what iconic characters we will be able to see recreated in the future.