Recent research around neural representations, such as NeRF (Neural radiance fields), has shown immense potential in storing high-quality scene and lighting information into compact networks. The major limitation of this explicit NeRF is the prohibitive computational cost from excessive network evaluations along each view ray for real-time rendering applications that require dozens of petaFLOPS.
Researchers from Facebook Reality Labs, USA and Graz University of Technology, Austria, have figured out a way to make neural representations practical for interactive and real-time rendering while sticking with the tight memory budget. They are especially interested in enabling large scale synthetic content in movie quality within an efficient render time period.
In this paper, the research team made some significant contributions with depth oracle neural radiance fields (DONeRFs). Apart from all this, DONeRF reduces the inference costs by up to 48x compared to NeRF when conditioning on available ground truth depth information.
They propose a new way to reduce the costs of evaluation in neural rendering. The idea is that an oracle network guesses where samples are needed along with view rays, while a shading network uses those predictions and places small numbers on top with guidance from the oracle for color input.
The research team presented a design and training scheme to provide sample locations for the shading network. The oracle uses filtered, discretized target depth values which are readily available in synthetic content. It learns to solve a classification task rather than estimate them directly.
A non-linear transformation was used to handle large, open scenes more efficiently than previous approaches. Sampling the shading network should happen with warped space and not linear space so that different frequencies can be captured both fore and background, which is something the current approach cannot do.
The research team can render high-quality, real-time neural scenes with the same tight memory budget that a NeRF has. This allows for a better quality of small scenes and significant improvements in large ones while reducing computational costs by 24–98x.
KeyPoints:
- DONeRF works in real time on single GPU
- DONeRF: Depth Oracle Net. + ray accumulation net
- DepthOracle predicts candidates on each ray
- DONeRF achieves a speedup of 20x-48x, same quality
Paper:https://arxiv.org/pdf/2103.03231.pdf
Github: https://depthoraclenerf.github.io/
Project: https://depthoraclenerf.github.io/
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
