Semantic retrieval had been failing with 401s while keyword fallback kept the assistant looking healthy. The first embeddings backfill exposed that production was authenticating with a literal environment-variable name.
September 23, 20265 min readSaytu team
Semantic retrieval had been off in production for as long as we had it. The system still answered questions. That was the problem.
We found it while running a new embeddings backfill for the first time. The embeddings endpoint returned 401. Until that run, nothing in the product had forced us to ask whether the semantic path was actually succeeding. The failure was not in the embedding model or the retrieval algorithm. It was one environment variable. Our .env contained:
EMBEDDINGS_API_KEY=$OPENROUTER_API_KEY
Next.js expands that reference when it reads the file. Docker Compose's env_file does not. It passed the literal string through instead. Once that value was already present in process.env, the .env loader did not replace it.
So the production container had been authenticating to the embeddings endpoint with a dollar sign and a variable name. The endpoint rejected it. And the rest of the retrieval system kept going.
If the whole assistant had failed, this would have been easy to find. It did not. Pages were still available to retrieval. Keyword matching still returned results. The assistant could still answer from what that path found.
What disappeared was the semantic pass. Pages were embedded, scored, and never once matched through it. The visible symptom was not an error screen. It was a keyword result in a place where a better semantic result should have appeared. That is a much harder failure to notice.
From the outside, retrieval was alive. The system returned knowledge. Responses arrived. A merchant could ask a question and get an answer. The missing thing was the path nobody could see.
We had built a system where one retrieval method could stop working and another method could make the whole pipeline look healthy. The fallback was doing its job. It was also hiding the failure.
The thing that finally exposed the problem was not a shopper complaint. A console.warn had been added around the embeddings failure in the previous change. When we ran the backfill, that warning made the 401 visible. That was enough to stop treating the retrieval result as a model-quality problem and follow the failure to authentication.
The lesson for us was not that keyword fallback was wrong. Removing the fallback would have turned a degraded retrieval path into a broken assistant. That is worse. The mistake was allowing fallback to become indistinguishable from normal operation.
A RAG pipeline can keep returning plausible context after one of its retrieval paths has stopped contributing anything. Looking only at the final answer cannot tell us which path produced it. In this case, it could not even tell us that semantic retrieval had disappeared.
There was no cosine threshold to tune. There was no prompt rewrite. There was no chunk-size experiment. The semantic retriever was not producing bad matches. It was failing before it had a chance to produce any.
We changed how environment references are resolved so the embeddings credential that reaches the container is the credential itself rather than the name of another variable. That took no ranking change. The same retrieval code could finally run as written.
This distinction matters because a bad answer from RAG does not automatically mean ranking is bad. Sometimes the ranking code never participated in the answer at all. Without visibility into that boundary, we could have spent time tuning the part of the system that was not running.
Fixing the credential did not make old vectors automatically useful. A vector only counts when it was made by the embedding model the current retrieval path expects. Rows synced before vectors existed, or before the embedding model changed, remain invisible to that semantic pass until they are rewritten.
So the same change carried a backfill path for synced rows, page chunks, and merchant FAQs. It skips rows already current, which lets us run it again without rebuilding everything that is already on the expected model.
The important part was that semantic retrieval becoming available again was not the end of the repair. The stored data had to agree with the retriever that was now asking for it. A working endpoint with stale vectors is another version of a system that appears healthy while a retrieval path quietly contributes less than expected.
This was the part we got wrong. We had treated the assistant continuing to answer as evidence that retrieval was working. It was only evidence that some retrieval was working. Those are not the same thing.
Keyword retrieval was strong enough to conceal the absence of semantic retrieval. That made the failure softer for shoppers and harder for us to detect. The system needed both properties: graceful degradation when embeddings fail, and a clear signal that degradation has happened. Before the warning, we had the first without the second. The backfill run finally showed the difference.
A model API failure is usually obvious. There is no answer, or there is an error to follow. Retrieval can fail more politely. One source can disappear. One candidate path can return nothing. One embedding request can fail while another retrieval method still supplies enough context for the model to produce fluent text.
That fluency is not evidence that the full pipeline ran. Our semantic path had been gone, and the system had enough remaining machinery to hide that fact.
So the finding from this change was not that semantic search beats keyword search, or that every RAG system needs a particular retrieval strategy. It was narrower. We had a retrieval path that could die without making the product look dead. The first backfill run caught it because we finally had a signal at the point of failure.
Once we could see the 401, the bug stopped looking like retrieval quality and became what it actually was: the container was authenticating with $OPENROUTER_API_KEY instead of the key. That one warning told us more about the health of the RAG pipeline than a working answer ever had.
