AI that reproduces only the video of a piano performance

A research team at the University of Washington developed an AI that reproduces the sound by viewing a video of a piano playing directly above the keyboard.

In order to reproduce the tone by viewing the video without audio, you must first remember which key on the keyboard reproduces which pitch. However, if the strength of the hitting or the length of the touch cannot be analyzed in detail, it is impossible to reproduce the expressive power by playing. Named Audeo, this AI reproduces the performance by converting the pitch, length, and intensity of the pitch, sound length, and strength, etc., from nuances into a form that can be interpreted by a digital synthesizer.

The goal is to reproduce videos that Audi has not seen yet to play quality music that is not different from the original song. As a result of showing, interpreting, and playing the first video on Audio, the research team confirmed the original song with an 86% recognition rate for the SoundHound app, which searches for the original song from humming. Originally, the recognition rate in video voice was 93%. However, for the 7% difference, further research is expected to reduce the error.

The research team hopes that this research will provide new ways to interact with music. For example, in the future, the Audi can be played live simply by photographing human hand movements with a virtual piano camera. In addition, Audi can help in a new way to teach piano class students how to play by placing a camera on a real piano. If Audi recognizes the opposite video through video calls such as zoom and makes the sound played on the speaker in close proximity, it may be possible to teach detailed performance techniques without making a loud sound even if it is impossible to face the student directly due to the Corona 19 disaster. . Related information can be found here .