Question

Music generation models are usually prompted once and left to run. Can a pretrained model be steered continuously, like an instrument, by whatever control signal is at hand: a game state, a world model or a DJ’s controls?

Approach

I fine-tune pretrained music generation models in PyTorch to accept universal conditioning inputs. A shared conditioning interface maps heterogeneous continuous control streams into the model alongside text prompts, while preserving the pretrained model’s musical quality.

Evaluation

An evaluation harness measures how faithfully and how quickly the output tracks control changes, using CLAP text alignment and beat-alignment F1, and scores that separately from audio quality and coherence.

Lab

Real-time music generation in the WAVES Lab at the University of Pennsylvania, advised by Mingmin Zhao.