Six policies against two and a half thousand metafields, one shared limit, and a refund policy the assistant never saw. Nothing was broken, and nothing reported it.
September 21, 20263 min readSaytu team
A store had six shop policies and roughly two and a half thousand product metafields, and for a while the assistant could not see a single one of the policies.
Nothing was broken. No sync had failed, no row was missing, no switch was off. The policies were in the database, correct and current, and they were never reaching the model. The reason is one of the least interesting lines of code in the system, and it is a good illustration of how retrieval quietly goes wrong.
It is tempting to think of retrieval as one step: score everything, take the best. It is two, and the first one is not clever at all.
Before anything is scored, a set of candidates is fetched from the database, and only four of them will survive to reach the model. There has to be a limit on that fetch, because scoring is not free and a large store has an enormous amount of text. So: take some number of rows, then rank them.
The original limit was one number shared across every source type. Take up to N rows, from anywhere, then rank what came back.
Here is what that assumes, without saying so: that a random-ish selection of rows is representative of what a store knows.
It is not, because source types do not produce rows at the same rate. Shop policies are a handful per store — refunds, shipping, terms, privacy, and one or two more. Product metafields are emitted by the sync at up to ten per product, so they scale with the catalogue. Two hundred and fifty products is two and a half thousand metafield rows against six policies.
With one shared window and no ordering, the window filled with whatever the database offered first. The sync writes products and their metafields early. So the candidate set arrived almost entirely metafields, the ranking did an excellent job of choosing the best metafields, and the refund policy was never in the room. (The ranking had its own ordering problem at the time, which is a separate post and a separate fix.)
The failure has no symptom you could search for. The assistant answers. It sounds fine. It simply answers a question about returns without having read the return policy, which is the one document written specifically to answer it.
So we gave each source type a guaranteed number of candidate seats, and no type can take another's.
The numbers we picked are not uniform, and the asymmetry is the point:
We removed no source. A small, high-intent type simply cannot be crowded out by a large one any more, whatever the catalogue looks like.
A store with more than about a hundred help pages still loses the tail: the quota slices before anything is ranked, so page one hundred and one is not beaten, it is never considered.
We wrote that into the code as a known ceiling with the upgrade path attached — rank pages before slicing, if answer quality ever shows it mattering. It has not yet, and doing it now would mean scoring a hundred extra candidates on every question to fix a case nobody has hit.
Naming it in the file is the cheap half. The expensive half is only worth buying when a real store proves it needs it.
