In what may be an early sign of a new technology for musical instrument playing and performance, Yamaha has demonstrated its artificial intelligence (AI) technology allowing a dancer, Kaiji Moriyama, to control a piano by his movements. The concert, held in Tokyo on 22nd November, 2017, was entitled “Mai Hi Ten Yu”‘ and was sponsored by Tokyo University of the Arts and Tokyo University of the Arts COI. Yamaha provided an original system, which can translate human movements into musical expression by using AI technology, as technical cooperation for the concert.
The use of gesture control to play instruments is not quite as far-fetched as it may seem. Already in the classical music world, the Canadian conductor and inventor, Dr Shelley Katz, has been using ‘robot’ traditional instruments as part of his innovative Symphonova concept. In that instance, orchestral instruments are also ‘played’ by gesture control from a conductor using traditional conducting gestures.
The AI adopted in the Yamaha system, which is now under development, can identify a dancer’s movement in real time by analysing signals from four types of sensors attached to a dancer’s body. This system has an original database that links melody and movements, and, with this database, the AI on the system creates suitable melody data (MIDI) from the dancer’s movements instantly. The system then sends the MIDI data to a Yamaha Disklavier player piano, and it is translated into music.
To convert dance movements into musical expression, the Yamaha Disklavier is indispensable, Yamaha says, because it can reproduce a rich range of sounds with extreme accuracy through very slight changes in piano touch. The actual instrument used was a special Disklavier in the concert was based on Yamaha flagship model CFX concert grand piano.