In this paper, the author presents softmax splatting for differentiable forward warping and demonstrated its effectiveness on the application of frame interpolation. The author also explains the current methods and discusses differentiable image sampling in the form of backward warping and how it has seen broad adoption in tasks like depth estimation and optical flow prediction. While most of the work has been done for backward warping, forward warping has been ignored partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way.
This paper proposes softmax splatting to address work on the forward warping which could be a paradigm shift along with effectiveness on the application of frame interpolation. In the paper, the author explains by considering given two input frames, then forward-warping the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. Further, the author talks about using a synthesis network to predict the interpolation result from the warped representations. The paper’s softmax splatting allows the team during experimentation to not only interpolate frames at an arbitrary time but also to fine-tune the feature pyramid and the optical flow. Through this paper, the author states that their synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.
Introduction to TensorFlow for Artificial Intelligence, Machine Learning, and Deep Learning [Advertisement]
Paper: https://arxiv.org/pdf/2003.05534.pdf
Github: https://github.com/sniklaus/softmax-splatting
Details: http://sniklaus.com/papers/softsplat
Asif Razzaq is the CEO of Marktechpost, LLC. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over a million monthly views, illustrating its popularity among audiences.
