A source with twenty stored chunks could mean either complete or truncated. We started recording the pre-cap count so cut, complete and unknown became three measurable states instead of one guess.
September 23, 20266 min readSaytu team
Twenty chunks did not mean we had the whole document. We had been treating them as if they did.
Our knowledge pipeline puts a cap on how many chunks from a long document we keep. The cap is twenty. That limit is intentional: at some point a source has to stop expanding into more stored context.
The mistake was not the cap. The mistake was that we stored no record of whether the cap had actually been reached. A document that naturally divided into exactly twenty chunks looked identical to a much longer document that had been cut down to twenty. Both ended up with twenty stored chunks.
One meant: we have the whole thing. The other meant: part of this document is missing, and the assistant cannot answer from anything beyond the part we kept. We could not tell which was which. So whenever we called a document complete from that count, we were guessing.
It is easy to look at a source sitting exactly on a limit and assume it was probably truncated. That is still an assumption. A document can genuinely divide into exactly twenty chunks.
The reverse mistake is worse: seeing twenty stored chunks and treating that as a healthy, complete source because nothing explicitly says it was cut. We had built a state that could support neither conclusion.
The database told us how many chunks survived. It did not tell us how many chunks the whole document would have produced. Without the second number, the cap erased the distinction we cared about.
That mattered because this is not a cosmetic status in an admin screen. The cap is a hard boundary on what the assistant can know from that source. If material falls beyond it, retrieval cannot recover that material later. It was never stored for retrieval in the first place.
We wanted the interface to say that when it was true. First, we needed the system itself to know.
The change was small: before applying the cap, we count how many chunks the whole body would produce. We store that count beside the chunks we keep. Now the two values can be compared.
If the full count is above the stored count, the source was cut. If the counts are equal, we have positive evidence that the source fit. And if the old source has no count at all, we do not infer either answer. We call it unknown.
That third state was the important one. It would have been easy to make the new field useful only for future documents and treat older rows as complete until proven otherwise. That would make the screen look cleaner, but it would turn missing measurement into a positive claim. We chose not to do that.
After the change, coverage has three real answers. A measured cut means we know the source lost chunks. A matching count means we know the complete body fit inside the stored set. No measurement means we do not know. Before this, those states collapsed into one suspicious count.
That distinction changes what the product is allowed to say. For a measured cut, we can tell the merchant that chunks were dropped and that the assistant cannot answer from material past the last stored part. For a source that fits, we can say so because we measured both sides. For an older source with no measurement, we do not issue a clean bill of health. We leave the gap visible.
That is less satisfying than turning every row green, but a knowledge screen that exists to show what the assistant can actually read cannot repair uncertainty by hiding it.
At first glance, it might seem enough to store the full count only when truncation happens. That recreates the same ambiguity in a different form. If the field exists only for a truncated source, then a missing field can mean either "this document fit" or "this document predates the measurement." Those are not the same thing.
So we write the count for every page. An equal count is evidence. An absent count is absence of evidence. Keeping those separate is what lets the system distinguish complete from unknown without inference.
The same rule applies to malformed historical metadata. The field lives in mixed metadata, so a value that is not a usable number is treated as absent rather than trusted. We would rather lose a status than manufacture one from data we cannot interpret.
Shipping the new write path only fixes documents written after the change. Everything already stored still has the old ambiguity.
We already had a chunk-identity backfill that walks the sources with their original body available, so we extended that path to calculate and stamp the full count as well. It can do that without calling a model. But until the production backfill is actually run, those existing pages remain guesses. The code does not pretend otherwise.
That part matters because schema changes often create a dangerous illusion: the application knows how to record a fact now, so the interface starts behaving as if historical records contain it too. They do not. The old rows only become measured when we measure them.
We also made the write path fail as one unit. If the provider fails and no chunks are written, we do not write a count beside them. A count on its own would look like knowledge about a retrieval artifact that does not exist.
So the guard exercises the real write path: a long body records that it was cut, a short one records that it fit, an unchanged re-embed preserves the measurement, and a body that later grows past the cap changes its status on the next embed. The measurement follows the thing it describes.
A lot of RAG discussion focuses on what happens after retrieval: whether the model should answer, abstain, cite, or ask for more evidence. Our problem happened earlier. Before retrieval could decide whether the evidence was enough, our ingestion path could remove part of a document without preserving the fact that anything had been removed.
No prompt can reconstruct that distinction. No confidence score can tell whether chunk twenty was the natural end of the document or simply the place where storage stopped. We had to measure it before the information disappeared.
That changed our definition of a healthy source. Healthy no longer means "the stored chunk count does not look suspicious." It means we have evidence that what we kept equals what the source produced. Everything else keeps its real name.
Unknown.
