Real Time Control of Virtual Orchestra by Recognition of Conducting Gestures

Mert Mermerci1,  Emile Pascoe2,  Fredrik Edström3,  Hedvig Kjellström1,4
1 KTH Royal Institute of Technology, Sweden    2 SMASH Studios, Sweden    3 IVAR Studios, Sweden    4 Swedish e-Science Research Centre, Sweden
arXiv · 2026
arXiv Dataset
Teaser figure

Figure 1. A real-time gesture control interface is presented, where the user can steer the pace of a virtual symphony orchestra.

Abstract

We present a museum installation in a 180° dome theater, which gives the museum visitor the experience of conducting a symphony orchestra. We have pre-recorded a short music piece performed by a professional orchestra. This recording is played back in the dome with the visitor standing in the conductor's position. The visitor's gestures are captured with a vision-based skeleton tracker, steering the recording playback pace via a gesture recognition module that translates the gestures into a time control signal. This is sent to a playback module that plays the recording in the dome at the corresponding speed. The gesture recognition module is based on a hierarchical LSTM network, trained with recorded sequences of multiple conductors with different level of expertise conducting the same recording. The system is evaluated with a quantitative study of the estimated timing accuracy, a user study evaluating the musical realism and usability of the real-time control, and a field study to evaluate the performance of the entire system with real museum visitors.

Video

Advertisement Video in the Wisdome Dome Theater

Promotion video clip for the opening of the museum installation on September 7, 2025.

BibTeX

@misc{mermerci2026realtimecontrolvirtualorchestra,
  title         = {Real-Time Control of a Virtual Orchestra by Recognition of Conducting Gestures},
  author        = {Mert Mermerci and Emile Pascoe and Fredrik Edström and Hedvig Kjellström},
  year          = {2026},
  eprint        = {2604.27957},
  archivePrefix = {arXiv},
  primaryClass  = {cs.HC},
  url           = {https://arxiv.org/abs/2604.27957}
}