When every score ties, the ranking is a coin flip

The assistant answers out of a store's own material: product descriptions, FAQ entries, pages the merchant pointed us at. Choosing which four pieces of it go into any one answer is a ranking problem, and for months that ranking had a bug no test would have caught. The output was never wrong. It just was not always the same.

Ties are the normal case, not the edge case

Part of the ranking is keyword scoring. A candidate earns points for the query terms it contains, and the scores are small integers over a modest number of candidates. Put those together and ties are not an edge case, they are the normal state of affairs. On a two-word question, half the shelf scores exactly the same.

When scores tie, the comparator returns zero and the order that survives is whatever order the candidates were already in. That order came from the database, and a database is not obliged to hand back documents in the same order twice. It usually does. Then an index gets rebuilt, or the collection is written to, or the plan changes, and it does not.

So the shopper's answer gets assembled out of a set that can differ between two identical questions, and nothing anywhere reports the change. No error. No warning. A slightly different answer to the same question on a different day.

The fix is deliberately not an improvement

The comparator now falls through to the record's own identity:

if (a.score !== b.score) return b.score - a.score;
return a.tiebreak < b.tiebreak ? -1 : a.tiebreak > b.tiebreak ? 1 : 0;

The tiebreak is the row's id as a string. Not its length, not its recency, not how well its title matches, not any of the half-dozen signals that would plausibly rank a tie better than an arbitrary string does.

That was the decision, and it was the harder half. Every one of those alternatives is a retrieval quality change: it moves which material reaches the model, it can be better or worse, and the only honest way to ship one is to measure it against a set of questions with known good answers. Identity order makes no claim at all. It is arbitrary, it is stable, and its entire job is to stop the ranking from being a coin flip.

Any deterministic choice beats a coin flip. Picking a better one is a separate decision that has to be measured, and the harness that would measure it cannot function while the ranking is still flipping coins. So: determinism first. Quality second, on purpose, with numbers.

The same rule had to go in twice

There is a second place ties matter, and it is easy to miss.

Before ranking, candidates are fetched under a per-type quota — take so many of this kind, so many of that. That query does its own ordering. If the ranking is deterministic but the fetch that feeds it is not, nothing is fixed: the set that arrives at the comparator differs between runs, so the comparator sorts a different set deterministically, which is the same non-determinism moved one step earlier and made harder to see.

So the tiebreak lives in both places — the ranking, and the query that decides which rows the ranking ever sees. A guarantee about order has to hold everywhere the order is decided, or it holds nowhere.

Nobody could have reported this

There is no crash here, no empty result, nothing that looks wrong on the screen. A merchant who asks the tester the same question twice and gets two slightly different replies cannot tell a ranking that flipped a coin from a model that chose different words. They have no reason to report either, and we would have had nothing to go on if they did.

What we go by instead: a comparator returning zero is a claim that the two records are interchangeable. If one of them is going into the answer and the other is not, they are not interchangeable, and zero is a lie that gets settled by whichever row the database felt like handing over first.