Nine tied policy rows produced two different top-threes on consecutive calls. The fix was not a better ranking — it was making the ranking we already had give the same answer twice.
September 21, 20264 min readSaytu team
Nine tied policy rows returned two different top-threes on consecutive calls. The question had not changed. The stored knowledge had not changed. The retrieval rules had not changed. The only thing that differed was the order equally scored rows happened to arrive in.
That was enough to make the source set unstable, and an unstable source set makes every measurement downstream of it worthless.
Part of the ranking is keyword scoring: a candidate earns points for the query terms it contains. Those are small counts over a modest number of candidates, so ties are the normal case rather than the edge case.
When several rows scored the same, the comparator returned zero and the surviving order was whatever the database read had produced. A database is not obliged to hand back documents in the same order twice. The record cap then kept whichever tied rows arrived first.
So we could ask the same question twice and hand the four slots that reach the model different material.
The obvious repair is a deterministic tiebreak after the relevance score: sort by score, then by a stable identity. That settles the final order of the rows that reach the ranker.
It does not settle which rows reach it.
Before ranking, a per-type quota cuts a store's rows down to a candidate window. If the database read feeding that quota is unsorted, the candidate set itself changes between calls, and a perfect tiebreak in the ranking cannot recover a row that never entered the window.
So the same rule had to go in twice. Candidate queries sort by _id, which gives the read a total order. The ranking sorts by score and falls through to the same stable identity. The source set stops depending on incidental database order on both sides of the quota.
Determinism is a property of a pipeline, not of a stage. A deterministic stage downstream of an arbitrary one is still arbitrary.
The tiebreak uses identity on purpose, and we argued about it.
It does not prefer a policy over a metafield. It does not prefer the most recently synced source. It does not smuggle in a new notion of authority. Every one of those is a defensible idea, and every one of them changes which source wins for a reason other than reproducibility — which makes it a retrieval quality change, and the only honest way to ship one of those is to measure it against questions with known good answers.
What we did here is narrower on purpose: when two candidates have the same score, we want the same one every time. We are not claiming the winner is better. We are stopping a coin flip from wearing relevance as a costume.
It also has to be this way round. A harness that measures retrieval quality cannot function while the thing it measures is still flipping coins, because the noise swallows the signal. Determinism first, quality second, with numbers.
A stable tiebreak is useful to the runtime and meaningless to a person reading a retrieval explanation. A row did not become more relevant because its id sorted first.
So the explanation reports the tie itself: this candidate shared its score with another, and identity settled the position. It does not present the id as evidence of anything. A reviewer sees that a placement was arbitrary, instead of being invited to read meaning into a hex string.
An FAQ takes part in the tiebreak as well, even though it does not carry a normal row id. We keep that case away from the chunk panel, so the interface never offers a source-inspection path that opens an FAQ and finds nothing behind it.
The rule is pinned in the retrieval suite rather than left as a convention somebody remembers.
Five identical calls must return identical sources. A store over its per-type quota must offer the same candidate set twice. Those two checks cover both halves of the bug: unstable ranking among ties, and unstable membership in the window.
That is the right boundary for a fix like this. A better ranking policy may deliberately change which source wins later, and it should. But the same code, the same data and the same question should not change their mind because the database returned tied rows in a different order.
The failure was small enough to look harmless — nine rows, two calls, two top-threes. It was still enough that every retrieval comparison we might run would have inherited the uncertainty. We did not fix it with a new embedding model or a new reranker. We made the strategy we already had reproducible, before asking whether it was any good.
