Key results

  • 0.85

    Mean recall at 1% FPR

    Across 10 held-out generators, at a conformal-guaranteed false-positive rate.

  • 0.96 / 0.83

    Suno / Udio recall

    At the same operating point.

  • 0.11

    Feature baseline

    DSP and embedding features alone, at the same operating point.

Publications

Three public documents separate the empirical result, the reusable evaluation method and the deployment argument, without exposing the private architecture or training recipe.

  1. Benchmark report

    The empirical record: leave-one-generator-out performance, low false-positive operation, matched MP3 stress, and the limits of the clean result.

    Public research report, 5 pages, July 2026

  2. Evaluation protocol

    A model-agnostic procedure for generator-disjoint splits, held-out human calibration, matched transformation tests, and comparable result records.

    Public technical protocol, 5 pages, July 2026

  3. Deployment and limitations

    A risk-oriented account of intended use, prevalence, calibration drift, subgroup errors, review workflow, and production stop conditions.

    Public deployment note, 5 pages, July 2026

Why

In-distribution accuracy is the easy version of this problem. A useful detector has to transfer to generators it never saw, survive ordinary re-encoding, and avoid falsely accusing human musicians. Detector is aimed at licensing and review workflows where the cost of a false positive matters as much as recall.

Approach

The project is organised around robustness rather than a single accuracy figure. It uses complementary evidence from audio forensics and learned representations, but the feature definitions, model topology, training recipe, and combination logic remain private. The public methodology is deliberately more detailed: reference- and song-disjoint splits, leave-one-generator-out folds, matched transformation stress tests, held-out human calibration, independent out-of-corpus checks, and reporting at predeclared low-false-positive operating points.

Results

Across ten generator families, each held out from its training fold, mean recall is 0.85 at a conformal-guaranteed 1% false-positive rate. Suno recall is 0.96 and Udio recall 0.83 at that threshold, against 0.11 for DSP and embedding features alone. Transformation-stress results are tracked separately, because clean-audio performance does not establish real-world robustness. The public benchmark report documents the earlier June run and its protocol.

Things I got wrong

The first near-perfect results were rejected after controls showed that the models had learned source-corpus and codec shortcuts rather than a transferable distinction between human and generated music. One apparently excellent system then failed on a real out-of-corpus export. Those failures led to same-song comparisons, song-disjoint splits, a leave-one-generator-out benchmark, removal of non-transferable covariates, matched transformations, and independent checks beyond the development corpus. I publish only numbers that survive those controls.

Future work

The immediate work is to complete the expanded generator benchmark, strengthen decisions across multiple excerpts from a track, and test the low-error review procedure on a complete score bank. Beyond that, the limiting factors are larger licensed human corpora, broader generator coverage, tighter transformation testing, calibration across more musical domains, and a production workflow that gives reviewers useful evidence without overstating certainty.

Research standard

Generalisation

Does the result transfer to generators the system has never seen?

Leave-one-generator-out folds and independent out-of-corpus checks.

Robustness

Does the conclusion survive ordinary changes to the audio?

Matched re-encoding, compression, and domain-shift stress tests.

False-positive control

What error rate is acceptable when human work is at stake?

Held-out human calibration, low-error operating points, and genre-specific audits.

Public
Research question, evaluation protocol, aggregate results, limitations, and the failures that changed the methodology.
Private
Feature definitions, model topology, fusion logic, training recipe, weights, source code, and data lineage.

Timeline

  1. Runnable research pipeline

    End-to-end training, evaluation, low-error calibration, and local report generation became operational.

  2. First real-generator transfer

    The initial transfer test succeeded beyond the development data; transformation and false-positive controls followed.

  3. Same-song evaluation

    A matched-content test exposed a dataset-identity shortcut and forced a redesign of the evaluation protocol.

  4. Misleading near-perfect model rejected

    An out-of-corpus failure was traced to source covariates. The result and the non-transferable inputs were discarded.

  5. Song-disjoint retraining

    Stricter separation produced a more defensible clean benchmark and a clearer account of transformation sensitivity.

  6. Ten-generator benchmark

    The self-contained leave-one-generator-out run became the first result worth publishing; genre-conditional calibration met the electronic false-positive gate.

  7. Low-FPR improvements tested

    New feature and loss experiments did not beat the June model. Fisher segment aggregation and calibration-tail review did produce independent gains.

  8. Champion rerun partially complete

    Seven of twelve generators completed before the compute cap.