Choose a judge's threshold from people's decisions, not from another judge

Criba collects job postings, drops the ones that fail simple rules and sends the rest to a language model acting as a judge. The judge scores each posting from 0 to 100 against a person’s profile, and anything above that person’s threshold triggers an alert. At first the thresholds were 70 and 80, tuned while the judge was Gemini.

Then I wanted to switch the judge to Claude. The tempting move was to swap the model and keep everything else. I ran a small check first.

A flawed experiment

I took the 10 postings Gemini had scored highest and gave them to Claude Sonnet with the same prompt. Sonnet scored them 36.8 points lower on average, and on none of the 10 did the two judges agree on which side of the cutoff the posting fell.

As a comparison of models, that’s worth nothing. It was a single run over 10 postings, I didn’t keep the raw outputs, and the sample was biased by construction: the postings one judge scored highest are bound to come out lower under any second opinion, because some of those scores were high by luck. Zero agreements out of 10 may be more than regression to the mean would produce on its own, but without re-scoring the same 10 with Gemini I can’t separate the two.

The question was narrower, though: could the thresholds survive a change of judge? It was enough to stop me from assuming the scale had stayed put. A score from an LLM judge isn’t a property of the posting. It belongs to the posting, the prompt and the model together, so changing any one of them means every threshold built on it has to be measured again. I lowered the thresholds from 70/80 to 50/60, and I treated them as a stopgap, not as measured thresholds.

Choosing the threshold from decisions

Comparing two judges against each other can’t tell me whether either one matches what people want. People’s decisions can. Criba records every reaction to an alert, apply or pass, per person.

So I designed a threshold report that groups postings by score band and compares each band with what each person decided. It only suggests moving a threshold once a band has at least 10 answers, because below that the noise wins.

Its first result came in October, from the first person’s decisions. They applied to 1 of 12 postings in the 60s band, 5 of 25 in the 70s and 7 of 13 at 80 and above. The judge’s score does carry information about what that person wants, and their threshold of 60 was letting through two bands they mostly pass on. Under the 25% rule, the report suggested raising it to 80. I didn’t: those two bands held 6 of their 13 applications, and with 25 answers the 70s’ interval still crossed the bar. The threshold stays at 60, and the report reruns as answers accumulate. That’s the number the model comparison could never have given me: not whether two judges agree, but whether one of them agrees with the person it works for.