Ambient Music co-Player: Generating affective video in response to impromptu music performance

When a musical instrument player performs music, the accompanying visual information can have a significant effect on the performance. In this paper, we present an ambient music co-player (AMP) as a system that generates background videos in response to the impromptu performance of a single musical instrument player. The AMP system evaluates the performance to interpret the real player’s emotional impression and generates an influential video based on the results of the evaluation. The player tends to change their performance while being inspired by the generated video, further triggering the system to modify the video. The AMP system aims to establish an affective loop where the system continues applying stimuli to the performance of the real player. The final goal of this study is to make the system act as a “co-player” of the player and to amplify the quality of the player’s performing experience entirely through interactions between the two. By conducting a user evaluation, it was proven that the AMP system was able to inspire an amateur guitarist as the subject through affective video generation and to make his performance better than when playing alone.