A research team at the University of Zurich presented a method of shooting a 5,400 fps ultra-slow motion image in high quality using an Event Based Camera and a neural network developed as a model of the retina.
If you shoot 24fps video with a regular camera, you can take 24 photos per second. Slow-motion video is to be played back slower than it actually is if you shoot too many frames per second, such as 120fps and 240fps, and set the playback frame to 24fps or 30fps.
The event camera used by the research team is a camera that detects only the change in brightness of light hitting the sensor, unlike a general camera that quantifies and outputs the brightness of light hitting the sensor through a lens. The research team built a system that reconstructs frames from event camera data, saying that event cameras cannot obtain clear images like regular cameras, but have a wide dynamic range, no subject bias, and a delay time of a few microseconds.
The research team first developed an algorithm that manually reconstructs frames from event camera data. Then, using the simulated data, we trained a neural network that can reconstruct event data with the same algorithm and input the actual data.
The video shot with the event camera and reconstructed with AI is black and white, but it can capture the shape and movement of fine objects well while shooting at the same quality as the 240fps video shot with the Huawei P20 Pro.
Event cameras only recognize changes in brightness, so unlike ordinary cameras, they are not good at capturing the shape of objects clearly. But nonetheless, the picture quality is comparable to that of a video shot with an iPad.
Humans have two eyes, but cameras are basically monocular, making it difficult to recognize the depth of objects. However, the AI trained by the research team can also estimate object depth from event camera images.
According to the research team, the image quality of slow-motion images reconstructed with AI is 20% higher than that of general slow-motion images. If you shoot a video at a frame rate of 5,000 fps or higher, the amount of light entering the sensor is extremely reduced, making the lighting conditions difficult for general cameras. However, by using event camera data that reads changes in luminance, high-quality slow-motion video can be captured at a high frame rate.
Most of the cameras that can shoot at frame rates higher than 5,000 fps are expensive for business purposes. Experts expect that using such a technology may enable super slow-motion HDR video recording at an affordable price. Related information can be found here .