A retry handed one store the reply written for another. The key was a string the browser chose, used across a collection that spans every tenant.
September 21, 20264 min readSaytu team
One store's reply
A shopper sends a message, the model answers, and the reply is stored against a turn. If that shopper's connection drops mid-answer and the page retries, we would rather hand back the reply that was already produced than charge them a second time and make a second model call for the same question. So there is a lookup: given a turn, find the reply that was committed for it.
The turn id is a string the client supplies. Sit with that one for a second.
The reservation collection holds every store's turns, in one place, keyed by that id. The obvious lookup is the obvious one: find the committed reservation with this turn id, follow it to the stored message, return the body.
That lookup is wrong, and it is wrong in the worst available direction. Two stores can produce the same turn id — the id is opaque, the browser decides it, and nothing in the system forces one store's ids to be disjoint from another's. When they collide, the lookup happily returns the other store's reply: the text, and with it the product cards and the order card that were attached to it. One merchant's shopper reads an answer written about another merchant's catalogue, possibly with another shopper's order in it.
Nothing exotic is needed to trigger it. A client library that starts its counter at one. A timestamp with second resolution. A retry that reuses a constant. Any of those collide across tenants as a matter of course, because the ids were never meant to be globally unique in the first place. They were meant to be unique inside one conversation, which is all the client can see.
The fix is small and it is not clever. The lookup takes the tenant as well as the turn, both are required, and either one missing returns nothing:
if (!turnId || !tenantId) return null;Then the query names both. A replay is only ever a replay of this tenant's own turn.
The bug is invisible in every environment where it matters least. In development there is one store, so no two tenants exist to collide. In a test the ids are generated by the test, so they are unique by construction. On a real deployment it needs two stores to pick the same string on the same day, which is rare enough to look like it never happens and common enough to be certain that it eventually does.
What found it was not a report. It was reading the collection's shape and noticing that the key was global while the thing it identified was not.
Which gives us a rule we have been repeating to each other since:
An identifier supplied by a client scopes to that client, never further. It can be a key inside the conversation it came from. Used across a collection that spans tenants, it is a lookup by a string an untrusted party controls. Adding the tenant is not defence in depth. The tenant is the key, and the client's id is one component of it.
Two smaller things came out of the same read.
The stored body is returned exactly as it is, with no redaction pass on the way out. Every storefront row is written with a flag meaning "personal data was stripped before this was stored" — the write side of the three retention windows — so running the read-time redaction rule over it would blank every single replay — the flag says the work was already done, and doing it twice erases the result. What anonymisation actually stamps is a different field, and a reply that has been scrubbed is never replayed at all.
The reservation's index is still on the turn id alone. The query is correct now, but the index does not yet match the query, which means the right rows come back after more work than they should. That is written down rather than fixed here, because an index change is its own deployment with its own risk, and the correctness problem was the urgent half.
We went through every collection that holds rows from more than one store, and for each query against it asked one question: which fields make this answer unique? Not which fields are indexed, which is a different question with a tempting overlap. Which fields make it right.
It is slow and it is boring and it found two more places we had been lucky rather than careful.
