Despite the advances in imaging technology, image noise and restricted sharpness remain most critical factors for increasing the visual quality of an image. Noise can be linked to the particle nature of light, or electronic components may be introduced during the read-out process. A photographic editing process will then process the captured noisy signal via the camera image processor (ISP) and be enhanced, amplified, and distorted. Image blur may be caused by a wide range of phenomena, from inadvertent camera shake during capture, incorrect camera focusing, or sensor resolution due to the final lens aperture.
It is much easier to minimize the effects of noise and blur within a camera pipeline because the details of the sensor, optical hardware, and software blocks can be understood. However, improving noise and sharpness becomes much more challenging when given an image from a random and unknown camera where we lack the understanding of the camera’s internal parameters.
Most of the time, these two issues are inextricably linked. For example, while noise reduction tends to remove fine structures and unwanted details, blur reduction seeks to enhance structures and fine details. This interconnectedness complicates the development of image enhancement techniques that are computationally efficient enough to run on mobile devices.
A new Google research introduces an approach for camera-agnostic estimation, eliminating noise and blur in images. They paired their pull-push denoising algorithm with a deblurring method, called polyblur, to maximize computational efficiency. These components enable users to successfully enhance the quality of a multi-megapixel image in milliseconds on a mobile device.
To precisely process a photographic image and minimize the undesired effects of noise and blur, the type and level of noise and blur in the image must be identified. The first step in a camera-independent approach for noise reduction is to devise a mechanism for determining the degree of noise at the pixel level in any given image, irrespective of the device that made it. The noise level is defined as a function of the underlying pixel’s brightness.
The team sampled many small patches across the image to estimate the brightness-based noise level. After removing any underlying structure, they measured the noise level within each patch. Because this process is repeated at multiple scales, it is resistant to artefacts caused by image resizing, compression, or other non-linear camera processing operations.
The fundamental idea behind non-local denoising the image is that noisy pixels can be denoised by averaging pixels with similar local structures. However, these approaches require a brute force search for pixels with similar local structures. This incurs high computational costs because and makes them impractical for on-device use.
With effective information propagation across spatial scales, the algorithmic complexity is decoupled from the size of filter footprints thanks to effective information propagation across spatial scales.
Pull-Push Method: Reducing Noise Selectively
In the pull-push method, the initial step is to create a multiscale image representation in which a “pull” filter generates each consecutive level recursively (similar to downsampling). This filter uses a per-pixel weighing mechanism to mix existing noisy pixels based on patch similarity and predicted noise, lowering noise at each consecutive, “coarser” level. Only suitable pixels from higher resolution, “finer” levels are pulled and aggregated by pixels from coarser levels (i.e., lower resolution). Each merged pixel has an approximated dependency measure generated from the similarity weights used to create it in the coarser layers. Therefore, the combined pixels offer a simple, per-pixel characterization, per-level picture characterization, and local statistics.
Then the “push phase” merges the results from the coarsest level and iteratively generates finer levels. At any given scale, the “filtered” pixels are generated going from coarse to finer levels. At each level, the pixels are fused with coarser-level pixels using the corresponding reliability weights, considering the weighted average of same level pixels, reducing pixel noise, and maintaining the local structure.
This algorithm has linear complexity (with the number of images) to selectively filter similar pixels over a substantial neighborhood. Therefore, the pull-push approach has a low computational cost as compared to traditional non-local methods.
Polyblur: The deblurring method
We require a more computationally efficient method than existing methods for computing and removing blur with minimal user delay. At any point in a sharp image, the maximum value of the image gradient across all directions follows a specific distribution. According to the team, determining the maximum gradient value is efficient. It can generate a solid estimate of the blur strength in the given direction and can immediately recover the parameters that define the blur.
The team describes the need to solve a numerically unstable inverse problem (i.e., deblurring) to recover the sharp image given the estimated blur. The inversion problem becomes exponentially more unstable as the blur strength increases. As a result, they assume that the image is mildly blurred but not beyond practical repair.
To put it another way, this method deblurs an image by re-blurring it with the estimated blur kernel several times. Each (estimated) blur application corresponds to a first-order polynomial, and repeated applications (adding or subtracting) correspond to higher-order polynomial terms. This method is called “polybur” because it is extremely fast. As a result, it only requires a few blur applications, processing megapixel images in just a fraction second on a mobile device.
The new adjustments are integrated into the Google Photos image editor in two new adjustment sliders called “Denoise” and “Sharpen.” Users can now use them to improve the quality of everyday images captured with any capture device.
Paper: https://ieeexplore.ieee.org/document/7532702
Source: https://ai.googleblog.com/2021/06/take-all-your-pictures-to-cleaners-with.html
Tanushree Shenwai is a consulting intern at MarktechPost. She is currently pursuing her B.Tech from the Indian Institute of Technology(IIT), Bhubaneswar. She is a Data Science enthusiast and has a keen interest in the scope of application of artificial intelligence in various fields. She is passionate about exploring the new advancements in technologies and their real-life application.
