Material selection determines which items in a scene are made of the same material. Knowing which products are made from the same components is helpful for a robot that has to manipulate them while, for example, cooking. With this information, the robot would know to use the same amount of force, whether picking up a small pat of butter from a dark corner of the kitchen or a whole stick of butter from inside the brilliantly lighted refrigerator. Machines have a hard time with this since the way something looks can be drastically altered by factors like the object’s shape and the lighting.
Efforts of researchers at MIT and Adobe Research have partially resolved the problem arch. They devised a method that locates all instances of a specified substance in a picture, as represented by a user-selected pixel, and displays them. Their machine-learning algorithm is foolproof to the effects of shadows and illumination changes that can make the same material appear different, and the system works accurately even when objects alter in size and shape.
Although the system was taught using only “synthetic” data—generated by a computer that manipulates 3D environments to make many different images—it performs well in genuine indoor and outdoor situations it has never seen before. If a user selects a pixel in the first frame, the model may recognize things in subsequent frames constructed from the same material. This method can also be applied to films. In addition to its utility in robotic scene perception, this technique may also find a place in picture editing software or computational systems that employ visual cues to infer material properties. It might also be put to use in content-based online recommending systems.
All pixels representing the same material are difficult for current material selection methods to identify correctly. Some approaches include just whole items; however, even something as simple as a chair might have a variety of components made from different materials. While certain techniques call for a specific set of materials, such as “wood,” thousands of different kinds of wood exist.
Using a machine-learning strategy, researchers could examine every pixel in a picture in real-time to find the material similarities between a user-selected pixel and the rest of the image. For example, their algorithm can correctly detect similar regions in an image containing a table and two chairs, assuming the tabletop and chair legs are wood. The team had to get over some obstacles before they could create an AI technique that could learn to select related materials. To begin with, they were unable to train their machine-learning model on any preexisting dataset because none of them provided materials with labels granular enough for their needs. Roughly 50,000 photos and over 16,000 materials were randomly applied to each object in the researchers’ synthetic dataset of interior scenarios.
Application of Model
- Editing images: Many more options exist for modifying images now that we may select components depending on their materials.
- Advice is given after carefully reviewing the source information. Finding your way around a huge online data set, like a catalog of products, is a real pain. Researchers demonstrate a way through which a new dimension of exploration can be introduced into the dataset: material similarity.
- The technique is unaffected by changes in illumination or perspective. Generalization to genuine pictures and unseen materials from a fully synthetic data training set paves the way for novel uses.
- This approach fails in areas where direct cast shadows are particularly strong. Since straight shadows are so much darker than their surroundings, they convey relatively little about the subject matter.
In their studies, the team discovered that their model was superior to others at predicting which parts of an image held the same content. When comparing their model’s predictions to the ground truth—the parts of the image made of the same material they found that it was accurate within 92% of the time.
Improving the model to pick up on finer features of items in an image would be a great way to increase the precision of their method. The proposed method expands the available set of image selection tools, streamlines numerous editing processes, and supplies crucial data for subsequent operations like material detection and acquisition. Scholarly contributions that make this possible include the following.
- The first material selection method is suitable for natural images; it is unaffected by variations in shading and geometry.
- A novel query-based architecture was developed with inspiration from vision transformers to pick pixels based on user input.
- In this new, massive collection, synthetic HDR photos are paired with fine-grained material classifications for each pixel.
Check out the Paper. Don’t forget to join our 21k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com
Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone's life easy.