Automatically generate intermediate frames only before and after the video?

According to a paper published by the Google AI research team on the archive (arXiv), a pre-examination thesis release site, it has succeeded in creating an intermediate frame with only the first and last frames of the video using the newly developed AI.

Researchers at Google Research, Yunpeng Li, Dominic Doblek, and Marco Tagliasacchi have developed an AI that creates a plausible video intermediate frame with only the first and last frames of the video. . The developed AI is composed of a neural network referring to the visual cortex of an animal, and is composed of elements such as a 2D convolutional image decoder, a 3D convolutional generator, and a video generator. First, the image decoder maps the first and last frames to the potential space, and the potential representation generator learns the input frame information. Finally, the video generator generates the intermediate frames.

The research team tested the AI performance developed using three samples. The sample is a short image of 16 frames and has a resolution of 64 x 64 pixels per frame. With only the first and last frames extracted from this video, the AI was asked to create the middle 14 frames.

AI performed modeling 100 times per sample and revised the model, repeating the same process about 10 times. The experimental data said that it took about 5 days using NVIDIA Tesla V100, a GPU. When comparing the frame created by AI and the actual frame in the resulting three videos, there is not much difference between the frame extracted from the actual image and the frame created by the AI.

The research team said, “Imagine a system that can automatically create animations using manga frames, and it will certainly revolutionize the animation industry.” Although it is not at the level of the ability to create animation by putting manga frames in AI, further advances in AI will help the animation industry to simplify. Related information can be found here .