\documentclass[10pt]{article}
\newcommand{\DocShortTitle}{Evaluation protocol}
\usepackage{detectorpub}

\begin{document}
\doctitle
  {Evaluation protocol for AI-music detectors}
  {Generator-disjoint testing, human calibration, and reproducible reporting}
  {1.0}
  {Public technical protocol}
  {A model-agnostic procedure for evaluating cross-generator transfer at operationally meaningful false-positive rates. The protocol is intended for detector authors, auditors, and reviewers who need comparable evidence without requiring public model weights.}

\section*{Protocol in one page}

\begin{enumerate}
  \item Define the score interface before viewing test labels.
  \item Group generated audio by generator family and content by song identity.
  \item Hold one entire generator family out of training per fold.
  \item Keep all related clips from a song in one partition.
  \item Reserve a verified-human calibration set, disjoint from training and test.
  \item Select the threshold for the declared target human FPR on calibration data only.
  \item Freeze the threshold and score the held-out generator plus human test set.
  \item Repeat for every generator family; report each fold, mean, and worst case.
  \item Repeat with transformations applied symmetrically to both classes.
  \item Publish manifests, counts, hashes, configuration, and a machine-readable result record.
\end{enumerate}

\disclosure

\section{Scope and threat model}

The target is binary review triage: estimating whether an audio item is consistent with AI generation under a stated benchmark. The protocol evaluates \emph{cross-generator generalisation}, not provider attribution. It assumes no active adversary in the core run; adversarial edits belong in a separately labelled stress suite.

The primary failure being controlled is a false positive on verified human music. Secondary failures include memorising a generator family, leaking song identity across splits, learning source-corpus differences, and exploiting transformations applied to only one class.

\section{Required inputs}

\subsection{Scorer contract}

The submitted detector must implement a deterministic callable that maps a waveform and sample rate to one scalar:
\[
  f(\text{waveform}, \text{sample rate}) \rightarrow s \in [0,1],
\]
where larger values indicate stronger AI-generated evidence. Any internal segmentation and aggregation must be fixed in advance and described. The evaluator must not require access to features, weights, or architecture.

\subsection{Manifest fields}

Each item should carry, at minimum:
\begin{itemize}
  \item stable item and song identifiers;
  \item class label and generator family where applicable;
  \item artist or creator identifier when available;
  \item source corpus, acquisition path, and license status;
  \item audio duration, format, sample rate, and transformation condition;
  \item cryptographic hash or equivalent integrity identifier.
\end{itemize}

No item may enter a public bundle if its redistribution rights are unclear. A manifest may point to data without redistributing it.

\section{Split construction}

\subsection{Generator-disjoint folds}

For generator families $G=\{g_1,\ldots,g_K\}$, construct fold $k$ so that all outputs from $g_k$ are excluded from model training and development. The fold test set contains outputs from $g_k$ and a held-out human test set. Repeat until every family has served as the unseen family once.

Hyperparameters may be fixed globally before the LOGO run. They must not be tuned on the held-out family's labels or scores. If retraining for every fold is infeasible, that limitation must be declared and the result must not be called LOGO.

\subsection{Content and creator separation}

All excerpts, encodes, covers, or transformations derived from one song belong to one split. Where artist identity is available, use artist-disjoint human partitions. Content-matched human originals and AI covers are preferred because they reduce the chance that genre, composition, or mastering differences stand in for the target label.

\subsection{Optional temporal holdout}

A temporal holdout may be added to estimate drift, but cannot replace LOGO. Training and calibration timestamps must precede the temporal test window, and generator-version changes should be recorded where known.

\section{Calibration}

Let $H_{cal}=\{h_1,\ldots,h_n\}$ be a verified-human calibration set and let $s_i=f(h_i)$. For a target FPR $\alpha$, choose a conservative upper-tail empirical quantile of the human scores. One finite-sample form uses the order statistic
\[
  t_\alpha=s_{(\lceil(n+1)(1-\alpha)\rceil)},
\]
with scores sorted in ascending order and the index clipped to the available range. The precise quantile convention must be recorded in code and metadata.

Calibration data must be disjoint from training, hyperparameter selection, and final testing. Reusing the test set to choose $t_\alpha$ invalidates the operating-point claim.

\Needspace{9\baselineskip}
\subsection{Sample-size warning}

At $\alpha=0.01$, the decision is governed by the extreme human-score tail. With only a few hundred calibration examples, one or two observations can move the threshold materially. Report $n$, the selected threshold, and a binomial confidence interval for realized test FPR. Do not describe a nominal 1\% target as an observed or guaranteed 1\% error rate.

\subsection{Conditional calibration}

If human score distributions differ across prespecified groups such as genre, estimate a threshold $t_{\alpha,g}$ within group $g$. Groups must be defined without test-label optimization. Always report group calibration and test counts, realized FPR, and the recall trade-off. If a group lacks enough data, fall back to the global threshold and disclose the limitation.

\section{Primary metrics}

For every held-out generator, report:
\begin{itemize}
  \item ROC AUC;
  \item recall at target human FPRs, minimally 1\% and 5\%;
  \item generated and human sample counts;
  \item realized human test FPR at the frozen threshold;
  \item confidence intervals or bootstrap intervals where defensible.
\end{itemize}

Across folds report the unweighted mean and worst case. A sample-weighted mean may be added but may not replace them. Accuracy and F1 may appear only as secondary metrics because both depend on class prevalence and an operating threshold that may not match deployment.

\section{Transformation stress tests}

Every transformation condition must be \emph{matched}: the same operator and parameters are applied to human and generated items. Recommended ordinary-use conditions include lossy codec export, sample-rate conversion, gain change, short excerpting, and common mastering operations. Each condition receives its own calibrated threshold unless the experiment explicitly tests threshold transfer.

Report clean and transformed results separately. Do not average them into one number. Preserve the content split when creating transformed derivatives, and cache transformation parameters in the manifest.

\section{Optional extensions}

\subsection{Calibration-tail audit}

Inspect the highest-scoring verified-human calibration items after the model and threshold are frozen. The audit may identify mislabeled data or a vulnerable subgroup, but any exclusion rule created from the audit must be rerun on a fresh calibration set. Post-hoc cleaning without re-evaluation is not valid evidence.

\Needspace{10\baselineskip}
\subsection{Multiple-excerpt decisions}

When a track produces several segment scores, declare the aggregation rule before evaluation. Report both segment-level and track-level performance, the number and duration of excerpts, dependence assumptions, and the failure behavior when only one excerpt is available.

\section{Submission record}

A reproducible result package should contain:
\begin{itemize}
  \item a unique system name and version;
  \item repository commit and environment lockfile;
  \item immutable manifest hashes and split seed;
  \item calibration policy and target FPRs;
  \item transformation definitions;
  \item per-fold machine-readable metrics;
  \item a plain-language limitations statement;
  \item disclosure of unavailable code, weights, audio, or licenses.
\end{itemize}

Reject a submission if it omits sample counts, reports only accuracy or F1, overlaps calibration and test data, uses cross-source classes without matching or controls, or publishes a mean without the worst-performing generator.

\section{Minimal result schema}

\small
\begin{tabularx}{\linewidth}{@{}l Y@{}}
\toprule
Field & Meaning \\
\midrule
\texttt{system\_id} & Immutable detector/version identifier \\
\texttt{source\_commit} & Evaluator and configuration commit \\
\texttt{fold.generator} & Entire generator family excluded from training \\
\texttt{n\_human}, \texttt{n\_generated} & Test sample counts \\
\texttt{calibration.n}, \texttt{target\_fpr} & Human calibration count and declared target \\
\texttt{threshold} & Frozen score threshold \\
\texttt{auc}, \texttt{recall\_at\_fpr} & Fold metrics \\
\texttt{condition} & Clean or exact matched transformation \\
\texttt{manifest\_hash} & Integrity link to evaluated items \\
\bottomrule
\end{tabularx}
\normalsize

\section*{References}
\small
S. Fullerton. \emph{AI-Music-Detection: public benchmark and evaluation framework}. GitHub repository, commit \texttt{4143b39}, 2026. \url{https://github.com/sonnyfully/AI-Music-Detection}

A. N. Angelopoulos and S. Bates. ``A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.'' \emph{arXiv:2107.07511}, 2021.

S. Bates, E. Candes, L. Lei, Y. Romano, and M. Sesia. ``Distribution-Free, Risk-Controlling Prediction Sets.'' \emph{arXiv:2101.02703}, 2021.

\vfill
\sourcefoot
\end{document}
