An AI judge that reads only 8% of the postings

The problem

Pulling from 13 job boards means seeing the same posting several times, once per site. Most of them don’t fit the profile either, and you only find out after reading.

I wanted an AI model to do that reading. Sending it every posting would be slow and expensive, and plenty of them fail a simple rule. So the model comes last, where rules run out.

Rules first, model last

Before any model call, Criba removes duplicates in four passes. The first compares the source’s own ID and the second the canonical URL. The third looks for near-identical titles within the same employer. The fourth checks postings with no employer against the rest of the batch.

It then filters in two phases, before and after fetching each posting’s details. That change made one board usable: before it, 1 posting out of 969 from that board survived.

The database shows the result. Of 40,015 unique postings, 3,210 (8.0%) reached the judge. Deduplication and rules handled the rest without spending a model call.

The judge’s contract

The judge gets batches of 20 numbered postings and answers in a fixed structure. A Pydantic model validates it with 4 required fields and no defaults, so a response missing a field fails validation.

Each verdict maps back to its posting by number. If it can’t be matched, it gets dropped and Criba doesn’t guess where it belonged. I would rather lose one verdict than attach one posting’s score to another.

How I measure the judge

To measure whether the judge matches what each person wants, I use their decisions as labels. Criba started out serving one person and now serves two, each with their own keywords and threshold in a file of their own. Both share one channel where they react to postings, and each reaction counts for whoever made it.

On top of those labels I built a threshold report. It groups postings by score band and compares each band with what each person decided, apply or pass. It needs at least 10 answers per band before it suggests anything, and a band stays above the threshold only if at least 25% of it was applied to.

The judge started on Gemini; switching it to Claude Sonnet moved its scale, so I lowered the thresholds as a stopgap before choosing them from decisions, as I explain in this post.

The rate rises from one band to the next, so the judge’s score does track what that person wants, although with 13 to 28 answers per band the margins overlap. Only the band at 80 and above clears 25%, which is why the report suggests raising their threshold from 60 to 80.

I didn’t. The bands from 60 to 79 hold 6 of their 13 applications, and at 80 none of those would have been alerted. With 28 answers the 70s’ 18% has a wide margin, roughly 8–36%, which still crosses the bar. Reading a few extra alerts costs less than missing an application, so the threshold stays at 60 and the report reruns as answers accumulate.

Why not 70, then? The 60s band stays under 25% even at its upper bound, 22%, so 70 would also satisfy the rule. Raising it would save 22 alerts and cost 1 of the 13 applications. With so few, losing one weighs more than reading 22 extra alerts, so I left it where it was.

The labels also exist only for postings that reached the channel, so the report can say a threshold should go up but not what is being missed below it.

What broke in production

The worst failures were the silent ones. In September, when a board rate-limited a search, the collector read the refusal as “no more results”, skipped every later search and ended the sweep with a green tick. I caught it because a full cycle with 25 keywords finished every source in 109 seconds, when that board’s search grid alone needs about 12 minutes. A rate limit is now treated as a refusal, and the next cycle resumes where it stopped.

In October an audit of production data showed that the rotating search on one board, the one whose postings survive the filters most often, had searched only the first person’s keywords, none of the second person’s. Each rotating search kept its place in memory, so every deploy sent it back to the top. Rotations now persist their place on disk, and both people’s keywords are interleaved.

Each of these got a written decision record with what broke, how it was detected and what changed.

How it is built

Hexagonal architecture in four layers (domain, adapters, application and interface) with a single composition root. Each cycle runs 15 collectors, built on httpx and Playwright, over 13 job boards and validates the data with Pydantic. Each person is alerted according to their own threshold. Data lives in PostgreSQL, in a container beside Criba on the same server. I moved it off Supabase in October, after full reads and nightly dumps used up the free plan’s egress.

Python 3.13 managed with uv, with mypy –strict relaxed only in the scraping adapters, and typed SQLAlchemy 2.0 with 14 Alembic migrations. CI triggers the deploy on the VPS over SSH, with a key restricted to a single command. The container runs as non-root, and the nightly backup keeps 7 daily and 8 weekly copies.

What gets checked before a deploy

I review every change before accepting it, and whatever can become a rule goes into CI. Four architecture contracts fail the build if an import crosses the layers. Six checks have to pass before anything ships, including strict types and a 95% branch-coverage floor that can only go up.

The deploy never migrates the database without a fresh backup, and when it finishes it verifies that the healthy container runs the commit that was built. In production, a per-cycle heartbeat tells a process that is making progress from a hung one that Docker would still call healthy.

Limits

The threshold reading rests on one person’s 63 decisions in the bands they get alerts for; for the second person only the lowest band can be read. A decision isn’t an outcome either: nothing after it is recorded, so I can say which alerts people act on, not where that leads.