Detector
Cross-generator forensic detection of AI-generated music.
Research record
- Benchmark reportFive-page public report covering the LOGO benchmark, fixed low-FPR operating points, matched MP3 stress, and human-domain audit.
- Evaluation protocolFive-page model-agnostic protocol for splits, calibration, transformations, metrics, and reproducible submissions.
- Deployment and limitationsFive-page risk note on intended use, prevalence, drift, review workflow, monitoring, and stop conditions.
- Research implementationReproducible training, scoring, calibration, and report-generation system. Source code and implementation details are private.
Mean clean AUC
Song-disjoint leave-one-generator-out evaluation across ten generators.
Mean recall at 1% FPR
Aggregate recall across unseen generators at a threshold calibrated to a 1% human false-positive rate.
Suno / Udio recall
At the 1% human false-positive operating point in the strengthened clean run.
Electronic realized FPR
Observed on a separately evaluated electronic-human calibration slice; not extrapolated beyond that domain.
Publications
Three public documents separate the empirical result, the reusable evaluation method, and the deployment argument. They document the evidence without exposing the private detector architecture or training recipe.
Benchmark report
The empirical record: leave-one-generator-out performance, low false-positive operation, matched MP3 stress, and the limits of the clean result.
- Type
- research report
- Version
- 1.0
- Published
- July 2026
- Length
- 5 pages
- Record
4143b39
Evaluation protocol
A model-agnostic procedure for generator-disjoint splits, held-out human calibration, matched transformation tests, and comparable result records.
- Type
- technical protocol
- Version
- 1.0
- Published
- July 2026
- Length
- 5 pages
- Record
4143b39
Deployment and limitations
A risk-oriented account of intended use, prevalence, calibration drift, subgroup errors, review workflow, and production stop conditions.
- Type
- deployment note
- Version
- 1.0
- Published
- July 2026
- Length
- 5 pages
- Record
4143b39
Generalisation
Does the result transfer to generators the system has never seen?
Leave-one-generator-out folds and independent out-of-corpus checks.Robustness
Does the conclusion survive ordinary changes to the audio?
Matched re-encoding, compression, and domain-shift stress tests.False-positive control
What error rate is acceptable when human work is at stake?
Held-out human calibration, low-error operating points, and genre-specific audits.What is public
Research question, evaluation protocol, aggregate results, limitations, and the failures that changed the methodology.
What remains private
Feature definitions, model topology, fusion logic, training recipe, weights, source code, and data lineage.
Why
In-distribution accuracy is the easy version of this problem. A useful detector has to transfer to generators it never saw, survive ordinary re-encoding, and avoid falsely accusing human musicians. Detector is aimed at licensing and review workflows where the cost of a false positive matters as much as recall.
Approach
The project is organised around robustness rather than a single accuracy figure. It uses complementary evidence from audio forensics and learned representations, but the feature definitions, model topology, training recipe, and combination logic remain private. The public methodology is deliberately more detailed: reference- and song-disjoint splits, leave-one-generator-out folds, matched transformation stress tests, held-out human calibration, independent out-of-corpus checks, and reporting at predeclared low-false-positive operating points.
Results
The June 28 benchmark covers ten generator families, each held out from the corresponding training fold, and reports mean clean AUC 0.993 with mean recall 0.87 at a 1% human false-positive operating point. Suno recall is 0.97 and Udio recall is 0.89 at that threshold. A separate genre-specific human audit measured 0.0% realised false positives on its electronic-music slice, with only a 0.5-point recall cost on Suno and Udio. Transformation-stress results are tracked separately because clean-audio performance does not establish real-world robustness. A newer expanded run remains incomplete, so it is not used as the headline.
Things I got wrong
Several early near-perfect results were rejected after controls showed that the models had learned source-corpus and codec shortcuts rather than a transferable distinction between human and generated music. One apparently excellent system then failed on a real out-of-corpus export. Those failures led to same-song comparisons, stricter split hygiene, removal of non-transferable covariates, matched transformations, and independent checks beyond the development corpus. I publish only numbers that survive those controls.
Future work
The immediate work is to complete the expanded generator benchmark, strengthen decisions across multiple excerpts from a track, and test the low-error review procedure on a complete score bank. Beyond that, the limiting factors are larger licensed human corpora, broader generator coverage, tighter transformation testing, calibration across more musical domains, and a production workflow that gives reviewers useful evidence without overstating certainty.
Timeline
- Runnable research pipeline
End-to-end training, evaluation, low-error calibration, and local report generation became operational.
- First real-generator transfer
The initial transfer test succeeded beyond the development data; transformation and false-positive controls followed.
- Same-song evaluation
A matched-content test exposed a dataset-identity shortcut and forced a redesign of the evaluation protocol.
- Misleading near-perfect model rejected
An out-of-corpus failure was traced to source covariates. The result and the non-transferable inputs were discarded.
- Song-disjoint retraining
Stricter separation produced a more defensible clean benchmark and a clearer account of transformation sensitivity.
- Defensible ten-generator benchmark
The self-contained LOGO run reached 0.993 mean AUC and 0.87 mean recall at 1% FPR; genre-conditional calibration met the electronic false-positive gate.
- Low-FPR improvements tested
New feature and loss experiments did not beat the June model. Fisher segment aggregation and calibration-tail review did produce independent gains.
- Champion rerun partially complete
Seven of twelve generators completed before the compute cap. Suno and Udio were missing, so the June 28 result remains the published headline.