Differentiable rendering is a novel field that helps to calculate the gradients of 3D objects and allows them to get propagated through images without 3D data collection and annotation. Rendering in computer graphics generates 3D scenes defined by geometry, materials, scene lights, and camera properties. Rendering is a complex process. Its differentiation cannot be uniquely defined; thus, a straightforward integration into neural networks is impossible. Differentiable rendering (DR) constitutes a family of techniques that tackle such integration for end-to-end optimization by obtaining useful gradients of the rendering process.
Nvidia and Aalto University introduce a modular primitive to provide high-performance primitive operations for rasterization-based differentiable rendering. The proposed modular primitive uses highly optimized hardware graphics pipelines to deliver better performance than previous differentiable rendering systems.
Significant highlights of the proposed approach are as follows:
- Efficiency: It helps in rendering the 3D scenes efficiently that are complex in terms of geometric detail, occlusion, and appearance in high resolution.
- Minimalism: It easily combines and works compatibly with the existing modern automatic differentiation (AD) frameworks such as PyTorch and TensorFlow.
- Freedom: It has the liberty to support arbitrary user-specified shading and arbitrary parameterizations of the input geometry. It does not commit to specific forms.
- Quality: It supports texture filtering operations with the required mipmap levels generated internally, without making any assumptions about texture content.
Fig: It shows the comparison between different differentiable rendering systems. Scalability is surface tessellation and image resolution performance, Flexibility to random shading support, and Antialiasing to geometric edge smoothing. Occlusion suggests that covered surfaces won’t affect the resulting image, and Gradients regards the gradients’ correctness in the rendered image, and GI denotes global illumination.
The proposed differentiable rendering systems provide custom, high-performance implementations such as
- Rasterizing (converting an image from a vector graphic format into a series of pixels, dots, or lines together making up the desired shape)
- Large numbers of triangles
- Attribute interpolation
- Filtered texture lookups
- User-programmable shading
- Geometry processing
Fig: A simple differentiable rendering pipeline with the above-mentioned primitive operations highlighted in red.Texturing gives a per-pixel texture coordinate vector, retrieves a value from a surface then places it in the output.
To evaluate the proposed method’s performance, meshes of varying triangle counts were selected from the ShapeNet database. Those were then rendered using both the proposed method and two benchmarks (Soft Rasterizer and PyTorch3D) at multiple resolutions. As a result of the experiments, the proposed method showed that it requires much less rendering and gradient time than others. Also, it offered better scalability.
Fig: Comparison showed in terms of rendering, gradient times, and speedup factors
Paper: https://arxiv.org/pdf/2011.03277.pdf
Github:https://github.com/NVlabs/nvdiffrast
Tanushree Shenwai is a consulting intern at MarktechPost. She is currently pursuing her B.Tech from the Indian Institute of Technology(IIT), Bhubaneswar. She is a Data Science enthusiast and has a keen interest in the scope of application of artificial intelligence in various fields. She is passionate about exploring the new advancements in technologies and their real-life application.
