Sketching is the most universally accessible way to convey a visual concept. In contrast, creating GAN models has traditionally required knowledge in deep learning and an extensive dataset of exemplars. Can sketching be used as a more practical means for generating new generative models?
In this research work, researchers from CMU and MIT present a method, GAN Sketching, for rewriting GANs with one or more sketches to make it easier for novice users. They do so by changing the weights of an original model in accordance with user sketches. This is done through cross-domain adversarial loss, which encourages the output of both models to match each other’s sketching style while preserving diversity even though they are different domains such as art vs architecture drawings.
This new generative model presented in this research paper will be able to create images with the same color, texture and background as user’s sketches. By using off-the-shelf models pre-trained on large scale data, they devise an approach to adjust some of these weights in order for it to match what the user sketched out. The cross-domain method encourages them to look like the sketch and preserve colors, textures, and backgrounds so other elements don’t get mixed together or lost when looking at their work from afar.
The research team has created several customized GAN models. They find that these modified models can be used to generate new samples, interpolating between two generated images, or editing a natural photograph. These customizations mean minimal user input which is useful because there’s no need to collect any more data through manual filtering and image alignment when one exemplar sketch from the user will suffice.
Although, this new method of GAN models from hand-drawn sketches is inspiring because it makes this process possible for novice users. However, the research team found out that their method does not work with all drawings; they could only create a model matching Picasso’s sketch and Attneave’s cat sleeping pose. There are plenty of improvements needed before this becomes reliable enough for everyone!
Paper: https://arxiv.org/pdf/2108.02774.pdf
Project: https://peterwang512.github.io/GANSketching/
Github: https://github.com/peterwang512/GANSketching [Codes Coming Soon!]
Asif Razzaq is the CEO of Marktechpost, LLC. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over a million monthly views, illustrating its popularity among audiences.
