Google recently announced ways to blur and replace the background in Google Meet for better focus on the person rather than the surrounding. The new features are powered by cutting-edge web machine learning (ML) technologies built with MediaPipe that work directly in the browser, without any extra steps like installing additional software. One of the main motives for developing these features was to provide real-time, in-browser performance on almost all modern devices. It accomplishes this by combining efficient on-device ML models, WebGL-based rendering, and web-based ML inference via XNNPACK and TFLite.
Overview of Web ML Solution
The new features of Meet are developed with MediaPipe, Google’s open-source framework. It helps building multimodal (for example, video, audio, any time series data), cross-platform (i.e., Android, iOS, web, edge devices) applied ML pipelines. It also powers ML solutions like on-device real-time body parts and poses tracking. MediaPipe’s web pipeline leverages WebAssembly that improves speed for compute-heavy tasks. At runtime, the browser converts WebAssembly instructions into native machine code that executes faster than the traditional JS (JavaScript) code.
The solution processes each video frame by segmenting the user from their background and computes a low-resolution mask. This mask is further refined to align it with the image boundaries. It is then used to produce the output video via WebGL2, with either blurred or replaced background.
Fig: WebML Pipeline: All compute-heavy operations are implemented in C++/OpenGL and run within the browser via WebAssembly.
Thus, Google Meet introduced a new in-browser ML solution for blurring and replacing the background. With this, ML models and OpenGL shaders can run efficiently on the web. The developed features achieve real-time performance with low power consumption, even on low-power devices.
Source: https://ai.googleblog.com/2020/10/background-features-in-google-meet.html
MediaPipe: https://mediapipe.dev/
Shilpi is a Contributor to Marktechpost.com. She is currently pursuing her third year of B.Tech in computer science and engineering from IIT Bhubaneswar. She has a keen interest in exploring latest technologies. She likes to write about different domains and learn about their real life applications.
