A control that appears to work and does not is worse than one that visibly fails. Why retrieval reads the switch rather than the rows, and why the inspector is the same code path.
September 21, 20264 min readSaytu team
A merchant turns off a knowledge source. They open the panel, switch "pages" off, and expect the assistant to stop answering from their help pages.
The question is what "off" means to the code that builds the prompt, and the wrong answer to that is the kind of bug that never gets reported. The assistant keeps answering. The answers are still good. They are just still coming from a place the merchant believes they shut.
The obvious implementation reads the rows. Each synced page has a status; turning the source off marks them; retrieval skips anything not marked live.
That works until a writer disagrees, and several do.
Approving a page captured from the storefront writes a row and sets it live, because that is what approving means locally. A write that sets a status is a visibility change, whatever the function around it is called, and we have been bitten by the other direction of that too. A disable that is interrupted halfway leaves the rows it had not reached still marked as synced. Any code path that writes a knowledge row without consulting the channel's switch produces a row that looks perfectly answerable under a source the merchant has turned off.
None of those is a bug in itself. Each writer is doing a reasonable thing with the information it has. The problem is that "is this source on?" was being inferred from a property of the rows instead of read from the place it is actually stored.
So we made retrieval read the switch. A source type the merchant has turned off does not reach the prompt, whatever its rows say about themselves. The rows are still there, still correct, still ready if it is switched back on. They are simply not consulted.
It is worth being precise about what the merchant would have seen, which is nothing.
They would not see an error. They would not see a wrong answer — the pages were accurate, that is why they were synced. They would see an assistant that carried on being useful, using material they had decided it should no longer use, possibly because the pages were out of date, or internal, or about a product line they had stopped selling.
A control that appears to work and does not is worse than one that visibly fails. There is no moment where anybody learns.
We keep an internal view that shows what retrieval did with a question: which candidates it considered, what they scored, and why each one that was dropped was dropped.
The thing that matters about it is that it is not a re-implementation. It calls the same function the shopper's question calls, with one flag set, so the chunking, the ranking, the cosine floor and the thresholds are not similar to the real ones — they are the real ones.
A separate ranking built for inspection would be easier to write and would drift within a month, and a probe that drifts is worse than no probe at all: it reports confidently about a system that no longer exists. Then somebody debugs a retrieval problem against a description of retrieval rather than retrieval.
Nothing about the shopper path knows whether anybody is watching. That is the only version of this that stays true.
Putting those two together is what the word is doing when we say the assistant answers from verified sources.
It does not mean a model checked the claim. It means every line that reached the prompt came from a row in that merchant's own knowledge, of a type they currently have switched on, and that the path which selected it can be replayed and inspected without re-implementing anything.
When nothing clears the bar, the honest outcome is that no knowledge block is built at all. An assistant with nothing to cite should say so, and the machinery here exists to make sure that situation is reachable rather than papered over with the fourth-best paragraph in the store.
