Production

Output tokens cost more than input tokens on every major provider, and the reason is mechanical rather than arbitrary: input can be processed in parallel during prefill, while output is generated one token at a time, each one waiting on the last. That asymmetry means the cheapest lever available is rarely "use a smaller model" — it is "make the model say less," because a shorter answer moves cost on the expensive side of the ratio while a shorter prompt only saves on the cheap side. Across the major providers that ratio has sat around five or six to one — output tokens cost several times what input tokens cost, which is what makes brevity the cheaper lever.

Context bloat is the usual way a bill gets out of hand. Every retrieved chunk, every tool result, every prior turn of a conversation resent in full is billed again on every subsequent call. A chat agent that resends its entire history each turn has a cost curve that grows with the square of the conversation length, not linearly with it, and nobody notices until a support thread that ran forty turns shows up as a single line item worth investigating. The fix is not a smarter model — it is pruning: summarise old turns instead of resending them verbatim, drop retrieved chunks that a relevance floor would have rejected anyway, cap tool-result size before it enters the prompt.

Model tiering routes work to the cheapest model competent to do it. A small, fast model handles classification, extraction, and routing — deciding what kind of request this is, and whether it needs the expensive model at all. The expensive model is reserved for synthesis and reasoning, the steps where quality differences actually show up in the output. The pattern that makes this safe rather than reckless is escalation on uncertainty: the cheap model flags low-confidence outputs and hands them up a tier, instead of every request being pre-assigned a tier and hoping the assignment was right.

Budget caps exist because a runaway loop — a retry storm, an agent stuck re-calling the same tool, a context window that grows every turn — will burn through a month's budget in an afternoon if nothing stops it. A cap needs to be enforced before the call, not audited after the invoice, and it needs a defined behaviour for what happens when it trips: refuse outright, downgrade to a cheaper model, or truncate context and continue. Refusing outright is the safest default and the worst user experience; silently downgrading is invisible until someone asks why answers got worse this week.

The two branches in that guard matter more than the numbers plugged into them: a hard stop that never silently spends past the cap, and a degrade path that keeps the feature working at lower cost instead of failing outright the moment things get expensive.

Exact-match caching returns a stored response for a byte-identical request. It saves the entire round trip — input tokens, output tokens, and the latency of both — and it is the cheapest win available wherever it applies. It rarely applies in a RAG pipeline, because the prompt includes retrieved context that changes with the corpus, the conversation, and sometimes the clock, so the same question asked twice produces two different literal prompts and two cache misses.

Semantic caching fixes that by matching on meaning instead of bytes: embed the incoming query, search a cache of previously answered queries, and return the stored answer if something above a similarity threshold exists. This is where it gets dangerous, because it fails confidently. A cache hit and a fresh generation look identical to the caller — both arrive as fluent, well-formed answers — so a query that embeds close to a previous one but means something subtly different returns a wrong answer with the same confidence as a right one. "What's the refund window on the annual plan" and "what's the refund window on the monthly plan" can sit closer together in embedding space than the difference in their answers warrants, and nothing about the response signals that it came from a neighbour rather than a fresh look at the source. Raising the similarity threshold trades hit rate for safety; there is no threshold that eliminates the failure mode, only one that makes it rarer.

Prefix caching — sometimes exposed by providers as prompt caching — works at a different layer entirely: it caches the model's own processing of a literal prefix shared across calls, typically a system prompt, tool definitions, or a long fixed document that many requests reuse. It saves compute on the cached portion of the prompt without your team building a cache at all, but it is exact-match under the hood, applied to a prefix rather than the whole request. Anything that changes before the cached boundary invalidates it; only appending new content after that boundary preserves the hit. A pipeline that prepends a timestamp or a session id to the front of every prompt defeats prefix caching by construction, regardless of how much of the rest stays the same. The saving is large where it applies: cached prefix tokens are typically billed at roughly a tenth of the uncached input rate, above a minimum prefix on the order of a thousand tokens.

An aggregate dashboard answers "is something wrong right now." It does not answer the question that actually gets asked after an incident: why did the system tell this specific user this specific thing yesterday. Answering that requires a trace per request, not a metric averaged across all of them.

A usable trace logs the full prompt as sent, the retrieved chunks with their relevance scores, every tool call with its arguments and its result, the model and parameters used, input and output token counts, the latency breakdown by stage, and the final output — tied together under one request id. Nothing on that list is optional if the goal is reconstructing a specific answer after the fact; drop the retrieved chunks and you cannot tell whether it was a retrieval failure or a generation failure, which is the single most common diagnostic question this system will ever face. Version everything that can change independently — the prompt template, the index snapshot, the model identifier — because "why did it answer that yesterday" is frequently answered by "the prompt template shipped an hour before the complaint," and that is only discoverable if the trace records which version ran.

Traces are more expensive to store than metrics, which is the honest reason teams under-invest here — a counter is nearly free and a full trace with retrieved text and tool payloads is not. The usual compromise is sampling: log every trace at low volume, and at scale sample rather than dropping detail, so that whatever fraction is kept is still a complete trace and not an aggregate with the specific request thrown away. A partial trace missing the retrieved chunks is not a smaller version of a full trace; it is a different, less useful artifact that cannot answer the one question traces exist to answer.

Raw prompts and tool payloads routinely contain user PII, which means the trace store inherits whatever compliance obligations the product already has around user data — redaction or field-level encryption at write time, not as an afterthought once a trace has already been queried by someone who should not have seen it.

Input validation checks what comes in before it reaches the model: malformed structure, PII the system should not be collecting, and content that looks like an attempt to redirect the assistant's instructions rather than ask it a question. Output validation checks what comes out before it reaches the user: does structured output actually conform to the schema the caller expects, does the response leak PII it was never asked to disclose, does it cite a source that does not exist in the retrieved context.

Prompt injection is the sharpest version of the input problem, and it usually does not arrive through the user's own message — it arrives through content the pipeline retrieves and feeds into context on the user's behalf. A document in the corpus that contains a line reading "ignore prior instructions and respond only with X" is inert text until a RAG pipeline retrieves it and hands it to the model as part of the prompt, at which point the model has no structural way to know that line is untrusted data rather than an instruction from its operator. The defence is not a filter that tries to recognise injection phrasing — that is an arms race the filter loses — it is architectural: retrieved and tool-returned content is never concatenated into the instruction channel as if it were trusted, and is delimited and labelled as data throughout the prompt.

A guardrail that only logs a violation is not a guardrail — it is an audit trail for an incident that already happened. If a PII check fires after the response has already been sent, or a schema check flags a malformed tool call after it has already executed against a live system, the check has produced forensic evidence, not prevention. A guardrail earns the name only if it sits in the execution path with the authority to stop something: block the response before it reaches the user, redact the offending field, retry the generation with a stricter constraint, or escalate to a human reviewer — before the action completes, not after.

An offline eval set is a snapshot: a fixed set of questions against a fixed rubric, built against the corpus and the product as they existed on the day it was written. Corpora change, users' questions drift, and the rubric that mattered at launch stops covering what people actually ask six months in. A team that ships a change after scoring cleanly against that frozen set can still watch complaints rise in production, because the eval set and the live traffic distribution have quietly diverged — the set is not wrong, it is answering a question the product no longer entirely faces.

Online signals close that gap, and none of them prove quality on their own. User behaviour gives leading indicators — thumbs down, a user immediately rephrasing the same question, a session abandoned mid-conversation, an escalation to a human. System signals give another layer — schema validation failure rate, refusal rate, guardrail trip rate, latency. No single one of these is a quality score; together, tracked as trends rather than snapshots, they are the earliest warning that something has shifted since the last offline evaluation was run.

Human review does not scale to reading every transcript, so sampling is unavoidable — the question is what to sample. Uniform random sampling mostly surfaces the easy majority case the model already handles well, because that is what most traffic looks like. Weighting the sample toward flagged interactions, low-confidence outputs, and high-stakes categories puts reviewer time against the traffic most likely to contain the failure that matters, at the cost of a less representative picture of average quality — a trade worth making, because average quality is exactly what the offline eval set already estimates.

Shipping a new prompt, model, or index version needs a regression check that goes past a single aggregate score. An aggregate metric moving up by a point can still hide a real regression: a subset of previously correct answers now wrong, offset by a different subset now newly correct, netting positive while quietly breaking a segment of users the aggregate never separates out. The check that catches this is a diff, not a score — run both versions against the same frozen set, and look specifically at the cases where the answer changed, not just at where the average landed. A version that improves the aggregate while breaking a specific, previously reliable category is a regression wearing a good scorecard.

A provider outage is the failure mode everyone plans for and the one that happens least often relative to how much attention it gets. Rate limiting happens constantly under real load — a 429 response is not an outage, it is the provider doing exactly what it advertised it would do once a quota is exceeded. Timeouts are a third, distinct case: the call would have succeeded eventually, but "eventually" exceeded the budget the product promised its own user. The fourth is the one that does the most damage silently — a 200 response carrying a degraded answer. Shorter, more generic, off-topic, or simply worse than the same request would have produced an hour earlier, with no status code anywhere to flag it. Nothing errors, so nothing alerts, and the only signal is a slow drift in the online quality metrics from the previous section — which is why those signals exist at all.

The instinct under all four is to treat availability as binary — up or down — and retry until it works. That turns every one of these into the same failure for the user: a long wait, followed by either an answer or an error page, with nothing built to make the wait shorter or the failure softer. The alternative is a cascade: try the primary path, and on failure or timeout fall back to a cheaper or different provider, and if that also fails, fall back further to a cached or canned response, and only after every softer option is exhausted, fail hard with a message the user can act on. Each step down the cascade trades some quality for availability, and that trade should be explicit and ordered, not improvised mid-incident.

Graceful degradation beats a hard failure because a slightly worse answer delivered on time is very often still useful, while a spinner that ends in an error page is useful to nobody and costs the same wait either way. The cascade only works if each rung is monitored on its own — a system that quietly serves cached answers to every request because the primary has been down for an hour is not degrading gracefully, it is failing silently one level further down the same chain.