Neural network taught to turn pictures into music

Dutch developers have created a neural network that can exhibit an artificial analogue of visual-sound synesthesia - the ability to correlate visual sensations with sounds. The algorithm consists of two parts, one of which encodes the image into a high-level representation, and the second decodes this representation into music.

A feature of the algorithm is that it studied independently without image-music pairs. The developers described the algorithm in an article on arXiv.org, and will also talk about it at the ICCVW 2019 conference.

In a broad sense, artists, photographers and designers use paintings and other visual works as a way of conveying information to other people. However, this method of delivering information does not work if the person looking at the picture has vision problems. At the same time, visual works transmit information in a variety of ways, for example, using the plot, form, color and other features, that is, they can be described analytically.

Maximilian Müller-Eberstein and Nanne van Noord of the University of Amsterdam have developed an algorithm capable of converting between images and music, and during training it does not require correlating images with music, and learns this independently using teaching method without a teacher.

The algorithm is built on the architecture of an auto-encoder. Such an algorithm performs the conversion from the source data to a hidden representation, which carries the basic information about the source data and allows you to restore them in a fairly similar form. Auto encoders consist of an encoder and a decoder. The peculiarity of such algorithms is that, as a rule, the encoder and decoder work with different data. For example, recently researchers from Google used this architecture to convert the musical sequence of any instrument into a drum part.

nanonewsnet.ru