The "golden set" is a moving target — and that's fine.

Why we stopped trying to freeze evaluation benchmarks, what we do instead, and the small piece of machinery that makes a moving target tractable.

Why we stopped freezing them

The fantasy version of evaluation goes like this. You write down 500 examples, you grade them carefully, you check them into the repo, and now you have a benchmark. Every change to the system can be scored against this fixed thing. You get to plot a line that goes up and to the right. You get to talk about statistical significance in meetings. It feels like science.

In practice, the frozen benchmark is a fossil within three months. The product surface has moved — new tools, new tenants, new failure modes. The model has moved — a new base model dropped, or you fine-tuned, or someone changed the system prompt. The world has moved — your users are now asking different questions than they were when you wrote the eval. The number is still going up and to the right. It just no longer tracks anything that matters.

So we don't freeze them anymore. The golden set is a moving target, and the trick — the whole job, really — is to move it on purpose.

Three kinds of drift

It helps to be precise about what's moving. We've found three drifts that need to be tracked separately, because they each call for a different response.

1. Distribution drift. Users start asking new kinds of questions. The eval set, drawn from old logs, is no longer representative. The fix is structural: keep a rolling sample from production weighted toward what's actually being asked this month.
2. Label drift. What counted as a good answer six months ago doesn't anymore. The product changed. The customer changed. The fix is human: re-annotate, on a schedule, with a clear changelog.
3. Capability drift. The model can now do things it couldn't before. The old benchmark saturates and stops discriminating. The fix is to retire saturated items and replace them with the next frontier — what the system is currently failing on.

The replacement process

The process is short, boring, and the part everyone wants to skip. We run it weekly on active engagements.

1. Pull 50 fresh samples from production, stratified by user segment and intent.
2. Label them with the current rubric. Note any time the rubric itself felt wrong.
3. Diff this week's labeled set against the prior month: which items have new "right answers"?
4. Retire any item where the system has been correct for four consecutive weeks. It's saturated; it's not telling you anything.
5. Promote anything you labeled as a near-miss into the active eval set.

None of this is clever. It's curation. And the team that does the curation — the same humans, week after week — is what gives the benchmark its meaning. The benchmark is a number, but the meaning is in the people who maintain it.

A benchmark without a steward is a number without a referent. The line on the chart is still going up. Nobody can tell you why. — internal handbook, §3.2

A small concession to numbers

"But how do we compare runs across time, then?" — the obvious objection, and the right one.

The answer is: we don't, much. We compare runs against this week's eval set, and we track which items moved. The interesting signal isn't the aggregate score; it's the slice. "Reranker change improved 7 of the 12 items tagged multi-hop and regressed 2 of 6 tagged refusal" is a real sentence that produces real follow-up work. "Score went from 0.71 to 0.73" is not.

For longer-horizon trend lines, we keep a small anchor set: 30 items, hand-picked, deliberately stable, re-graded only when the product semantics actually change. The anchor set is the only thing we treat as frozen, and we treat it the way astronomers treat a reference star — useful precisely because it doesn't move.

What actually works

A few things that have held up across engagements:

• Per-item scoring with traces. Every eval item carries the full trace of what the system did. The score is almost a footnote next to the trace.
• A changelog for the rubric. When a labeler changes their mind about what's correct, that change is a first-class artifact. Half of "model regressions" turn out to be unlogged rubric changes.
• A weekly meeting that looks at five regressions. Not five aggregate numbers — five actual failing items. The team that can do this every week is the team whose product gets better.

None of this looks like science. It looks like editorial work. We've made peace with that. The benchmark is a moving target because the thing it's measuring is a moving target, and pretending otherwise is how you ship a number that nobody trusts.

Building an eval harness from scratch? Write to us.