Evaluation
A layered defense, each catching a different failure
An LLM feature without evaluation is a demo, not a product. The hard part is that “answer quality” is many different things, is it grounded, is it relevant, did it retrieve the right fact, did it correctly refuse, and no single number captures all of them. So there are several layers, four core ones below, plus a routing evaluation and a prompt-optimization loop, each designed to catch a specific class of failure before it reaches a shopper.
The four layers
| Layer | What it measures | Runs where | Catches |
|---|---|---|---|
| CI eval gate | Does the pipeline retrieve the right fact, or correctly abstain? | Every push, offline | A broken retriever or abstain path |
| RAGAS | Faithfulness, answer relevance, context precision & recall | On demand, real providers | A model or prompt that reads well but is ungrounded |
| Drift monitors | Distribution shift in traffic, grounding, and refusal rate | Scheduled, live traffic | The world moving out from under a frozen model |
| Promotion gate | Is a new config actually better than the champion? | Before promoting to prod | Shipping a regression to Production |
Layer 1, the offline CI gate
Five fixtures over a small, neutral corpus (no domain vocabulary, so it runs on deterministic fakes with no API keys). Four are in-domain questions that must retrieve the right fact; one is out-of-domain and must abstain. A cheap lexical judge, did a retrieved context contain the expected keyword, or did it correctly abstain, needs no LLM, so it runs in CI on every push.
score 1.0 min 1.0 blocks the merge below 1.0
The gate is deliberately strict: any single failed fixture drops the score below the threshold and blocks the merge. The evaluation notebook demonstrates it working by breaking the corpus on purpose and watching the score fall, a gate you’ve never seen fail is a gate you don’t trust.
Layer 2, RAGAS on the golden set
The four canonical RAGAS metrics, implemented from scratch rather than pulled from a package, so the judged text is sanitized against prompt injection and the scores break down per language:
Faithfulness
Of the claims in the answer, how many are supported by the retrieved context? This is the anti-hallucination metric.
Answer relevance
Does the answer actually address the question, or does it wander?
Context precision
Of the chunks retrieved, how many were relevant? Measures the reranker.
Context recall
Of the facts needed, how many were retrieved at all? Measures recall.
The judge runs at temperature 0, and can be pointed at an independent, stronger model (JUDGE_MODEL) so the system isn’t grading its own homework. RAGAS needs the real providers and an ingested store, so it isn’t a CI gate, instead it feeds the promotion decision below.
Layer 3, drift monitors
Four monitors watch live traffic: query-embedding distance (cosine between the reference and current query centroids), retrieval-score PSI (population stability index of the top vector-hit score), confidence PSI (of the lexical-overlap gate), and feedback rate (the change in thumbs-down). The retrieval-score and confidence monitors are stratified by language. A monitor that trips exits non-zero so a scheduled job can alert. The point isn’t to page someone at 3am, it’s to notice when the questions people ask start drifting away from what the corpus can answer, which is the early signal that the content needs to grow.
Layer 4, the promotion gate
A “model” here is a serving configuration: the LLM, the embedder, the reranker, the prompt. Promotion from dev → staging → prod is not automatic and not decided by the offline smoke test. It works in two stages:
- Precondition, the offline CI gate must pass, proving the pipeline is wired.
- Quality gate, promotion is decided by the measured RAGAS score from the real providers, persisted and logged to MLflow. A candidate reaches Staging at 0.65 and Production at 0.78, and a Production candidate must also beat the current champion or it’s capped at Staging.
This layer started life broken, it scored a fake-provider pipeline that always returned 1.0, so every run “promoted to Production” regardless of the real model. An adversarial review caught it. It now refuses to promote at all if no real quality score was ever measured: you cannot promote on a quality you never measured.
Layer 5, the omni-agent routing evaluation
The assistant became a master orchestrator that routes every turn to a lane, stylist, care, complaint, answers, or escalation, before it answers. Routing is the new thing to evaluate: send a turn to the wrong lane and the best answer in the world is off-topic. So there is a labeled set of 350 shopper turns (domains/apparel_ecommerce/eval/routing.jsonl, curated synthetic and described honestly as such) and a harness that scores the router with no retrieval and no generation, so it runs in seconds and costs at most one small-model call per ambiguous turn. Every number below regenerates from one command, scripts/run_agent_eval.py --with-llm, and lands in evaluation/reports/routing_eval.json.
| Mode | Accuracy | Marginal cost |
|---|---|---|
| Deterministic (layers 0 and 1) | 81.6% | none |
| 8B small-model tie-break | 85.9% | ~$0.0005 |
| 70B large-model tie-break | 85.6% | ~$0.0006 |
Two decisions fell out of this, and both are in the numbers rather than a preference. The cheap deterministic router carries most turns at zero marginal cost and 100% escalation precision. And a 70B tie-break does not even edge out the 8B (85.6% vs 85.9%), while both lift the free layer by only a few points, which is the measured case for staying Groq-only instead of paying a frontier model to classify.
Layer 6, the prompt-optimization loop
The evaluation is not just a scorecard, it is a target. The routing eval found the small-model tie-break was guessing a specialist lane on ambiguous turns instead of deferring. A small OPRO-style loop, about eighty lines rather than a framework, took that exact prompt, let a model propose more conservative rewrites from the failing cases, scored each on a held-out split, and surfaced the winner for a human to promote. It lifted the tie-break from 73.9% to 79.5% on the held-out test, which flipped the 8B tie-break from hurting the routing to helping it. That winner was promoted by hand and is now the baseline the served prompt sits at, so re-running the loop today finds no further lift, the loop working as intended, not a regression. Measure, find the weak spot, optimize, re-measure, with a human on the promotion step so the system never rewrites its own served behavior. The whole story, with the two verification rounds that caught an overfit in the router cues, is on the decisions page.