NeRF - Google's (AI) project improving the quality of heavily noisy and underexposed photos
Ben Mildenhall presents NIRF in the dark, which stands for Neural Radiance Fields. This is a scene representation that can be used for high-quality novel view synthesis. In the video, a NIRF model reconstructed from approximately 20 input images of flowers is showcased. NIRF works well in scenes like this where there is minimal brightness variation and no visible image noise. One of the input images for the scene looks clear and sharp, without overexposed highlights or heavily shadowed dark areas. But what if we want to capture a scene that is much darker? A cell phone image lit only by candlelight demonstrates that while the phone has made an effort to produce a decent-looking picture, it does so at the cost of losing a lot of hidden detail. If instead we take a raw file and apply minimal post-processing, then brighten the result, we observe a lot of sensor noise but also reveal more detail in the dark regions.
Ben Mildenhall highlights a patch behind the wooden bowl, further brightening it to show lost details. Running a state-of-the-art deep denoising network on the raw image removes some noise but leaves many artifacts. His method, known as rawnerf, combines images taken from various camera viewpoints to jointly denoise and reconstruct the scene. By going back to the noisy image, we see just how effectively rawnerf removes noise. However, rawnerf is not just a denoiser, meaning we can vary the camera position to view the scene from different angles. In this way, rawnerf goes beyond standard capabilities, reconstructing the scene and rendering new views in a linear HDR color space.
This enables Ben to vary exposure by rescaling renderings in linear space before post-processing them. He can also replace the basic gamma curve with a more complex tone mapping algorithm that brings out detail in shadows. The linear color values allow for rendering synthetic defocus with accurate bokeh effects. Mildenhall demonstrates some of what rawnerf is capable of in nighttime scenes. For instance, there's a noisy outdoor scene illuminated only by street lamps. By focusing on this street sign, we can see that rawnerf enhances legibility by cleaning up enough noise.
Ben compares the output of rawnerf to that of a standard nerf trained on post-processed JPEGs, showcasing a significant difference. He illustrates that rawnerf recovers much more accurate color and detail throughout the scene. In the remaining segments, Ben presents various scenes where rawnerf excels at recovering low details and image quality, even in challenging lighting conditions. Despite the noise, rawnerf is able to recover the fine geometry of objects like fences and bicycles. As he concludes, Ben showcases a variety of HDR view synthesis effects created using rawnerf reconstructions. He emphasizes how effectively this technology can create atmospheric effects and dramatically affect contrast in darker areas.
Ben Mildenhall's video has achieved impressive statistics with 136,104 views and 2,393 likes at the time of writing this article, indicating high interest and quality content based on the latest advancements in graphics technology.
Toggle timeline summary
-
Introduction to NIRF and its capabilities.
-
Explanation of Neural Radiance Fields (NIRF) enhancing scene representation.
-
Demonstration of NIRF with flower images.
-
Optimal usage of NIRF in low-variation scenes.
-
Contrast with dark scenes captured under low light.
-
Limitations of typical cell phone cameras in low light.
-
Utilizing raw files to reveal hidden details in dark areas.
-
Introduction to rawnerf, a method combining multiple viewpoints.
-
Performance of rawnerf in noise reduction.
-
Rendering new views and varying exposure with rawnerf.
-
Exploration of rawnerf capabilities in nighttime scenes.
-
Comparison between rawnerf and standard NERF outputs.
-
Presentation of rawnerf's performance on noisy frames.
-
Showcasing rawnerf's ability to recover fine geometry.
-
Highlighting HDR effects achieved with rawnerf.
-
Explaining dynamic range handling by rawnerf.
-
Enhancing contrast in darker scene regions using tone mapping.
-
Addressing HDR challenges with bright stained glass.
-
Utilizing rawnerf for reflections in complex scenes.
-
Conclusion and thanks for watching.
Transcription
Presenting NIRF in the dark. NIRF, or Neural Radiance Fields, is a scene representation that can be used for high-quality novel view synthesis. This video is a rendering of a NIRF model reconstructed from about 20 input images of these flowers. NIRF works very well in scenes like this, where there's minimal brightness variation and no visible image noise. Here's one of the input images for this scene. We can see that it looks clear and sharp, with no overexposed bright highlights or heavily shadowed dark regions. But what if we want to capture a scene that's much, much darker? This is a cell phone image lit only by candlelight. My phone has made a good attempt at producing a decent-looking picture, but at the cost of throwing away a lot of hidden detail. If we grab a raw file instead, apply only minimal post-processing, then brighten up the result, we can see lots of sensor noise, but also reveal more detail in the dark regions. Let's focus on this patch behind the wooden bowl, brightening it up even further. If we run a cutting-edge deep denoising network on the raw image, it removes some of the noise, but leaves behind a lot of artifacts. Our method, called rawnerf, is able to combine images taken from many different camera viewpoints to jointly denoise and reconstruct the scene. Going back to the noisy image, we can see just how well rawnerf does at removing noise. And of course, rawnerf is not just a denoiser, which means we can vary the camera position to view the scene from different angles. Rawnerf goes beyond this. We actually reconstruct the scene and render new views in a linear HDR color space. This means we can also do things like vary the exposure by rescaling renderings in linear space before they're post-processed, or swap out the basic gamma curve for a more complex tone mapping algorithm that brings out detail in shadows. We can even take advantage of the linear color values to render synthetic defocus with accurate bokeh effects. Let's take a look at what rawnerf is capable of in some nighttime scenes. Here's a noisy outdoor scene lit only by street lamps. Focusing in on this street sign, we see that rawnerf cleans up enough noise to make it legible. We can compare rawnerf's output to a standard nerf that's been trained on post-processed JPEGs instead of on raw image data. This rendering was originally so dark we had to scale it up by 16 times to make this video. By comparison, rawnerf recovers much more accurate color and detail throughout the scene. In the remaining examples, we'll simply show a noisy test frame, the corresponding rawnerf rendering, and then a path of novel views. Despite the noise, rawnerf is able to recover the fine geometry of this fence and bicycle. This shiny stove shows how rawnerf inherits nerf's ability to model view-dependent effects. Similarly, in this scene we can observe the reflection of the street light in the road as the camera moves from side to side. We'll end with a showreel of HDR view synthesis effects created using rawnerf reconstructions. As mentioned earlier, the linear color values recovered by rawnerf allow us to render physically accurate defocus effects. Using viewpoint, focus, and exposure altogether creates an atmospheric effect that can bring attention to different regions of the scene. Rawnerf is able to handle scenes with extremely large dynamic range when trained on images taken at different bracketed exposure levels. We can apply a sophisticated local tone mapping pipeline to improve contrast in darker regions of a scene. This effect can heighten the drama of scenes with stark lighting conditions. Capturing bright stained glass windows from inside a dark church has always been a classic HDR challenge. Bright background light sources with varying colors can create pleasingly varied bokeh effects. Since rawnerf builds a full 3D reconstruction over a wide baseline of inputs, we can even focus on the bookcase reflected above the piano keys. Thanks for watching.