Google Art & Culture attempts to reproduce synesthesia with AI

Google has unveiled a special feature of Russian artist Wassily Kandinsky, known as the founder of abstract painting through Google Arts & Culture. Among them, new attempts to reproduce synesthesia are also being made.

Usually, one sense responds to a stimulus such as sight for color and shape, hearing for sound, and smell for smell, but in the world, there are people who have synesthesia, such as feeling color in letters and sounds, and feeling color and shape in taste and smell. Even Wassily Kandinsky can connect colors and shapes with sound and mood on his own. Having seen the music, he entered the path of abstract painting to express the sound in pictures.

Google analyzed the notes left by Kandinsky in cooperation with the French Georges Pompidou National Center for Arts and Culture to see how he felt when he was drawing his work. At the time, it revealed a function (Play a Kandinsky) to learn the transformer neural network specialized in language comprehension using music and to experience the sound contained in pictures. In addition, Kandinsky seems to have heard red as a violin and yellow as a trumpet.

The synesthesia mindset may vary from person to person, so it is not possible to confirm whether it really sounds like this, but it can be said to be an interesting attempt as a new method of interpretation of conversation. It may also help people who don’t have empathy to know synesthesia. Related information can be found here .