In this paper, the author presents softmax splatting for differentiable forward warping and demonstrated its effectiveness on the application of frame interpolation. The author also explains the current methods and discusses differentiable image sampling in the form of backward warping and how it has seen broad adoption in tasks like depth estimation and optical flow prediction. While most of the work has been done for backward warping, forward warping has been ignored partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way.
This paper proposes softmax splatting to address work on the forward warping which could be a paradigm shift along with effectiveness on the application of frame interpolation. In the paper, the author explains by considering given two input frames, then forward-warping the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. Further, the author talks about using a synthesis network to predict the interpolation result from the warped representations. The paper’s softmax splatting allows the team during experimentation to not only interpolate frames at an arbitrary time but also to fine-tune the feature pyramid and the optical flow. Through this paper, the author states that their synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.