Google Meet Introduces In-Browser Machine Learning Solution For Blurring And Replacing Background In A Live Video

0
1098
Source: https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

Google recently announced ways to blur and replace the background in Google Meet for better focus on the person rather than the surrounding. The new features are powered by cutting-edge web machine learning (ML) technologies built with MediaPipe that work directly in the browser, without any extra steps like installing additional software. One of the main motives for developing these features was to provide real-time, in-browser performance on almost all modern devices. It accomplishes this by combining efficient on-device ML models, WebGL-based rendering, and web-based ML inference via XNNPACK and TFLite.

Overview of Web ML Solution

Advertisement

The new features of Meet are developed with MediaPipe, Google’s open-source framework. It helps building multimodal (for example, video, audio, any time series data), cross-platform (i.e., Android, iOS, web, edge devices) applied ML pipelines. It also powers ML solutions like on-device real-time body parts and poses tracking. MediaPipe’s web pipeline leverages WebAssembly that improves speed for compute-heavy tasks. At runtime, the browser converts WebAssembly instructions into native machine code that executes faster than the traditional JS (JavaScript) code.

The solution processes each video frame by segmenting the user from their background and computes a low-resolution mask. This mask is further refined to align it with the image boundaries. It is then used to produce the output video via WebGL2, with either blurred or replaced background.

https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

Fig: WebML Pipeline: All compute-heavy operations are implemented in C++/OpenGL and run within the browser via WebAssembly.

Thus, Google Meet introduced a new in-browser ML solution for blurring and replacing the background. With this, ML models and OpenGL shaders can run efficiently on the web. The developed features achieve real-time performance with low power consumption, even on low-power devices.

Source: https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

MediaPipe: https://mediapipe.dev/

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.