Creativity knows no barrier and is found across all spectrums. However, most often, language is found to be a significant roadblock in conveying that creativity to the masses. TV shows and films are one arena where widespread translation is used to reach wider audiences. But the question that arises is, does that change the viewing experience of the audience? The answer might be different for different individuals, but it could possibly be insignificant given the fact that the AI Startup Flawless claims to have found a technology to override this language barrier. With the help of their Deepfake Dubs, the performance quality and the emotion would be retained, and the film or show, even when translated, would be as authentic as the original.
The Main Focus: Lip Movement
The key to retaining the performance here is precision and focusing on the single most important element: the mouth. The systems are first fed with the videos and the translations done by humans. Then the machine learning models developed by the company automatically create new lip movements that go in line with the translated speech fed earlier. Simultaneously, the actor’s head also changes complementarity to the new lip movements.
Aim of the Company
The startup aims to provide people with the originality of the performance to be enjoyed without having to notice or be distracted by any mismatched mouth movement. Moreover, they also claim to offer value for money because of their potential to scale. With the popularity of OTT platforms ever-increasing, these translations have become even more important to reach international markets. The cost of remakes can prove to be pretty extensive, and in contrast to that, these translations are pretty cheap.
Working of the Technology
These techniques for dubbing have been developed in three years by Flawless, and a lot of energy has gone into speeding up the production time. Manual retouching is also done, but it is only around 15 percent in comparison to automated dubbing. If there are some lines or scenes where the dubbing seems to be a little out, iterations can be done by the company, and the training data can be resubmitted to get another result.
The translations are not 100% efficient, but they do give out excellent results. With the help of demos provided by the company and testing them on native language speakers, it has been found that while there are a few glitches in the translation, they might not be perfect. Still, at the same time, it has been acknowledged that they are smoother than the traditional dubbing done on these pieces.
Appealing Filmmakers for Dubbing
Filmmakers are usually very selective in their work and retain the original cast’s magic and performances. Therefore, the AI algorithm being offered by Flawless may be looked upon with keen interest. Subtitles can prove to be a lot of work for most people who would rather avoid watching the film or show, and as stated earlier, remakes may not always be the most feasible option to reach people all across the world. Remakes would not be replaced per se by dubbing; translating will only be another option for filmmakers to look into before going in for a remake.
Value for Money and Recognition
The translations would not be as attractive as the remake; that is a given fact; however, what is important to note is that the cost of the dubbing will only be 2 percent the cost of the remake. Dubbing as such already is present in the international film landscape, but with more popular options, increased efficiency, and authenticity of the performances being maintained, it could reach newer heights and more audiences with minimal effort. An additional advantage could also be recognizing regional actors for their performances instead of only the conventional ones because of the language barrier.
The significant roadblock now for Flawless is to prove that the audience wants this new piece of technology. Launched just recently, the company has already managed to get a contract, the name of the client has not been disclosed yet. While we have proof that the technology is working well in its arena, commercially, it remains to be seen if it proves profitable.
Company: https://www.flawlessai.com/
Project: https://gvv.mpi-inf.mpg.de/projects/NeuralStylePreservingVisualDubbing/
Paper: https://arxiv.org/pdf/1909.02518.pdf
Video: https://www.youtube.com/watch?v=p0muYUlQcJM&t=4s
Source: https://www.theverge.com/2021/5/18/22430340/deepfake-dubs-dubbing-film-tv-flawless-startup
Amreen Bawa is a consulting intern at MarktechPost. Along with pursuing BA Hons in Social Sciences from Panjab University, Chandigarh, she is also a keen learner and writer, having special interest in the application and scope of artificial intelligence in various facets of life.
