3D computer graphics have come a long way since their inception in the 1950s. From simple wireframe models to photorealistic renderings, 3D graphics have become an essential part of many industries, including film, video games, architecture, engineering, and product design.
One of the key principles behind 3D graphics is the use of mathematical models to represent three-dimensional objects. These models are made up of vertices, or points in 3D space, and are connected by edges to form polygons. By manipulating these vertices and edges, artists and designers can create and shape 3D objects with precision.
It has been decades since we switched to 3D modeling from 2D in video games, and things have been improving since then. We have realistic-looking games with much more freedom of movement and intense gameplay.
Video games are just one example of 3D computer graphics. We can think of other examples as 3D rendering. You can understand what I mean if you have watched an animated movie such as Toy Story. Those characters are not real; they are all computer generated and animated. Moreover, with all this CGI going on in modern movies, there is not a single movie without the use of computer graphics.
3D graphics have come a long way in recent years, thanks to advances in hardware and software. Modern graphics processing units (GPUs) are capable of rendering complex 3D scenes in real-time, making it possible to create immersive experiences in games and virtual reality applications.
The advantage of having things in the virtual world is the capability of having almost unlimited customization options. You build something, and you can design and customize it however you want. Though, doing it manually can be cumbersome as you need to make sure every single point in your 3D mesh is properly edited. For example, if you want to change the shoes of your 3D avatar, you need to edit it point-by-point by yourself.
Researchers from the University of Chicago have come up with a brilliant solution to highlight certain parts of 3D objects, which could be utilized for many applications. From selective editing to controlled stylization, this approach eases the process. What is the name of this method, you might ask, and the answer is 3D Highlighter.
The task of identifying specific areas or regions on 3D meshes based on their meaning is known as semantic localization, and it has various practical uses. One example is the inclusion of semantic information in the creation of 3D models. A difficult aspect of this task arises when the 3D geometric data alone is not enough to accurately perform segmentation, for instance, in deciding where to place a shirt on a 3D human model with no clothes. This is exactly what 3D Highlighter is trying to solve.
3D Highlighter automatically localizes the semantic regions of the 3D shape based on the text description. So you could ask it to mark the ears of a rendered rabbit, and it will precisely do that for you. It takes a mesh as input, making it compatible with 3D modeling workflows and tools such as Blender or Unity.
The 3D Highlighter system has the ability to determine where to put seemingly unrelated items on a 3D shape, like a hat on a lamp. It is able to locate attributes that are not present on the shape geometrically, a process referred to as “hallucinated highlighting” in the paper.
It uses a neural network to achieve accurate highlighting. The network learns to map each point on the surface to a probability of belonging to the text-specified region. The probabilities predicted by the network serve as a “soft-selection operator,” blending the highlighter color onto the mesh. The network’s weights are adjusted by encouraging the CLIP embedding of the 2D renderings of the highlighted mesh to match the specified text. This allows the network to learn how to segment the object to comply with the text prompt.
To summarize, 3D Highlighter is a technique for identifying specific areas on 3D shapes based on their meaning (semantic localization). The localization is specified through a text description, which is easy to use, flexible, and not restricted to a particular training dataset. Check the links below for more information.
Check out the Paper, Code, and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.
Ekrem Çetinkaya received his B.Sc. in 2018, and M.Sc. in 2019 from Ozyegin University, Istanbul, Türkiye. He wrote his M.Sc. thesis about image denoising using deep convolutional networks. He received his Ph.D. degree in 2023 from the University of Klagenfurt, Austria, with his dissertation titled "Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning." His research interests include deep learning, computer vision, video encoding, and multimedia networking.
