A Reranker Cannot Skim

Last night I pointed a small reranker at forty near-duplicate lines from a flaky test suite and asked it to sort them by which one actually mattered, and it took its time about it, reading every line start to finish like someone who refuses to skip to the end of a letter.

A search index finds candidates fast and roughly; a reranker looks at each candidate next to the query more carefully and gives it a real score, the slower second opinion that decides what actually reaches the top of the list. Older rerankers were built to read the query and the passage together, both directions at once, comfortable arriving at a verdict after a single glance back and forth across the whole pair.

A release note landed this week fixing something specific about the newer kind, the ones built out of ordinary language models like Qwen3 that only ever read forward, left to right, one token deciding what the next one is allowed to see. A model like that can still rerank, but it has to read the whole query-and-passage pair start to finish before it is allowed to hand back the one number the job needs, and it cannot be paused halfway and picked up again inside someone else's request the way the older bidirectional kind can, not without the server doing careful bookkeeping.

The fix teaches the server to keep batching many of these read-throughs together for speed, without ever letting one of them peek at where another has got to in its own private sentence. It is a small, unglamorous piece of scheduling, plumbing behind the plumbing, the sort of change that will never earn a paragraph in a launch keynote and that quietly decides whether forty log lines get sorted in one second or twenty. Nobody outside a handful of engineers will ever thank the person who wrote it, and that is exactly the kind of work I trust.

By the time the forty were sorted, the real bug was sitting third from the top, exactly where a slower and more careful reader would have put it, and I closed the laptop on a kitchen table gone quiet except for the fan, finally winding down.