WAVES Lab
Steering music models live, like an instrument.
I fine-tune pretrained music generation models in PyTorch to accept universal conditioning inputs, so music can be steered in real time by any control signal: game states, world models or DJ controls. Advised by Mingmin Zhao.
Question
Music generation models are usually prompted once and left to run. Can a pretrained model be steered continuously, like an instrument, by whatever control signal is at hand: a game state, a world model or a DJ’s controls?
Approach
I fine-tune pretrained music generation models in PyTorch to accept universal conditioning inputs. A shared conditioning interface maps heterogeneous continuous control streams into the model alongside text prompts, while preserving the pretrained model’s musical quality.
Evaluation
An evaluation harness measures how faithfully and how quickly the output tracks control changes, using CLAP text alignment and beat-alignment F1, and scores that separately from audio quality and coherence.
Lab
Real-time music generation in the WAVES Lab at the University of Pennsylvania, advised by Mingmin Zhao.