NeRF 3D: How to Create Photorealistic Environments with AI

*With the collaboration of Javier Cantón.

What is NeRF 3D And How Does It Work

Neural Radiance Field (NeRF) is a technology that revolutionizes the representation of 3D spaces. Visualizing photorealistic environments from different points of view with a high level of data compression opens up new possibilities for different industrial sectors, ranging from digital twins to the creation of scenarios for the media sector, with faster content production.

Scenes can be rendered as a triangular mesh. In each visible part, a shader is made that makes the inference on the color of the output pixel. After rendering the mesh as a grid, the MLP multilayer learning algorithm is used to recreate the features and opacity of each point. New algorithms are emerging that improve the performance of processing these complex scenes.

Puddles are fun… jump in! Once again having fun with reflections and NeRF. Trained and rendered using NeRFStudio running nerfacto – @nerfstudioteam.⁣⁣
⁣⁣#NeRF #nerfstudio #nerfacto #neuralrendering #mirrorworld #montreal pic.twitter.com/zjf5IIjKbg

— Hugues Bruyère (@smallfly) November 27, 2022

NeRFs can be successfully reproduced on a variety of devices, as shown in the image below from the paper MobileNeRF:

Until now, the technology to achieve a similar result was photogrammetry: using many images, a three-dimensional representation was generated with greater or lesser detail. This technology has been superseded by the use of Artificial Intelligence, which is what NeRF is based on. The advantage of NeRF is that it can capture the elements of a scene volumetrically. This means that, unlike photogrammetry, it is able to correctly represent transparent or metallic objects with reflections. This would help the visualization of products that are now difficult to see because they use transparent plastic material or have metallic protection with reflections that photogrammetry does not adequately capture.

Plain Concepts’ Research team is already integrating this technology into Evergine, our industrial 3D graphics engine. The result can be seen in this video:

In another example, one of our colleagues used two images of the backyard of his house to recreate it and simulate a visit. To do this, he used Azure Machine Learning, COLMAP (a free tool, downloadable on Github, that compiles collections of images to build 3D objects), Nerfstudio (an API to simplify the NeRF design process) and mip-NeRF 360 models, suitable for scenes where the ‘camera’ has to rotate 360 degrees. The result is a static image that could well have been recorded by a drone or a camera from a crane.

Instant NeRF

Based on NeRF, NVIDIA has launched Instant NeRF, which converts photos into 3D scenes with the help of artificial intelligence. As can be seen in the following video, with four photos and a neural network it is possible to create a 360-degree static image.

The use of Artificial Intelligence is very important to achieve a scene reminiscent of those seen with Google Street View. As the NVIDIA video says, “AI is turning 2D photos into 3D scenes in seconds”.

NeRF Applications For Your Company

Digital Twins in Real Estate and Smart Cities

Among the possible uses of NeRF is the improvement of Digital Twins for building construction or the development of Smart Cities: from now on it is possible to achieve more detailed replicas that help to make better decisions.

Media and Communication

Within the media industry, more specifically in the news area, reporters could recreate the scene of a catastrophe, an attack, or an awards ceremony with photographs and NeRF, and then narrate with immersive technology what happened there. In addition, in other areas, such as entertainment programs, production companies are interested in NeRF to generate very realistic sets in just a few hours, which would replace the current 3D reproductions. This reduces costs in set construction and, at the same time, allows us to have a television with a much richer audiovisual component, unimaginable a few decades ago.

Mixed Reality Applications in Interior Design

In addition to recreating real scenarios with photographs alone, NeRF allows you to include virtual objects in these spaces. This helps to improve the experience in Mixed Reality applications; for example, if your company sells furniture or home décor objects, you could develop an application for users to add photos of their rooms, generate a video with them, and include recreations of your products, in order to check if they fit with the color of the walls, the size of the room or other furniture, and thus make a purchase decision.

At Plain Concepts, we are working so that, once a NeRF model is trained, it can be visualized in real-time and the audience can move through the space.

Other Applications

In addition, work is currently underway to develop dynamic NeRF techniques that would generate three-dimensional videos from photographs. Thus, with virtual reality glasses, we would walk in a real space recreated through these images. It could also be used to advance holoportation.

NeRF can be integrated with Point Cloud as a layer that is superimposed on this point cloud and allows more precise work to be done, such as space measurements. Therefore, this new technology also helps in RealityCapture processes.

If you want to continue exploring the possibilities of NeRF and Instant NeRF for your company, do not hesitate to contact us.