Real-time Neural Radiance Fields (NeRF)
Speeding up Neural Radiance Fields (NeRF) for real-time applications
Neural Radiance Fields has recently emerged as a groundbreaking technique for novel view synthesis, and 3D reconstruction. While the foundational paper on NeRF shows impressive rendering quality, the method is slow, taking dozens of seconds to generate a single image.
Since then, the number of papers related to Neural Radiance Fields has exploded, and a specific research direction focuses on making them faster at inference time, sometimes reaching an impressive 200 frames per second (FPS).
In this tutorial, we will review 5 major papers in that direction.
- KiloNerf.
- FastNerf.
- PlenOctrees.
- K-Planes.
- Instant NGP.
If you are interested in their implementation, I have also included links to YouTube videos implementing them from scratch, in 100 lines of code! If you want to access the code directly, they are available at that GitHub repository.
Before diving in, if NeRF is still vague to you, you may be interested in my story about the foundational paper, or my Udemy course about it.
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
The concept of KiloNeRF involves the storage of Multi-Layer Perceptrons (MLPs) in a voxel grid, which means that each spatial region is represented by its dedicated MLP. By modeling smaller regions, these MLPs can be significantly reduced in size compared to a standard NeRF. Consequently, the forward pass in KiloNeRF requires only a fraction of the floating-point operations (FLOPs) compared to the original architecture. As a result, KiloNeRF achieves a notable increase in speed.
FastNeRF: High-Fidelity Neural Rendering at 200FPS
FastNeRF leverages conventional computer graphics techniques and introduces a novel approach by decomposing the scene into a two-dimensional deep radiance maps. This maps can be efficiently stored and combined with the directions of rays to generate pixel colours. This groundbreaking research achieves a remarkable speed improvement, surpassing the original NeRF method by up to 3000 times.
PlenOctrees for Real-time Rendering of Neural Radiance Fields
PlenOctrees can be considered a comparable approach to FastNeRF. It involves caching components that can be queried and combined with ray directions to generate pixel colours. However, instead of caching components into planes like FastNeRF, PlenOctrees learns spherical harmonics coefficients, which have been extensively studied in the field of computer graphics. This utilization of spherical harmonics coefficients in PlenOctrees provides a distinct method for representing and reconstructing the scene.
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance
K-Planes is a concurrent research effort to FastNeRF, sharing a focus on decomposing space into planes. However, unlike FastNeRF, K-Planes directly optimizes the planes themselves instead of initially optimizing an MLP and then converting it into planes. This distinction sets K-Planes apart in its approach. While FastNeRF can be adapted for dynamic environments, the K-Planes paper demonstrates impressive results specifically in dynamic scenarios, showcasing its effectiveness in 4D reconstruction.
Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
Instant NGP is a revolutionized paper in the field of Neural Radiance Field, and was named a Best Invention of 2022 by TIME Magazine. Instant NGP replaces sequential, and thus slow MLPs with a leanable hash encoding that can be queried in O(1).
Instant NGP is a groundbreaking research paper that has brought a revolutionary advancement to the field of Neural Radiance Fields. It has garnered significant recognition and was named a Best Invention of 2022 by TIME Magazine. Instant NGP replaces sequential and thus slow MLPs with a learnable hash encoding. This encoding allows for efficient queries with a complexity of O(1), contributing to substantial speed improvements and enhancing the overall performance of the method.
I hope this story was helful to you. If it was, consider clapping this story, and do not forget to subscribe for more tutorials related to NeRF and Machine Learning.
[Full Code] | [Udemy Course] | [NeRF Consulting] | [Career & Internships]
