Decisions
Why these parts, and where I’d switch
Every choice here is a trade-off against constraints: this runs on a hobby budget, it has to work offline for a reviewer with no API keys, and it has to be swappable so it isn’t wedded to one vendor. Below is the reasoning, and the honest “if your constraints differ, switch to this” for each.
Embeddings & rerank, Cohere
Chosen: Cohere embed-v4.0 (1536-dim) for embeddings and rerank-v3.5 for reranking.
The reranker is the highest-leverage component in the whole retrieval path. Dense vectors get you a rough top-50; the reranker is what turns that into a precise top-8, and a good one covers for a merely-adequate embedder. Cohere’s rerank is strong, multilingual (the demo runs English and French), and cheap at this scale. Embeddings and rerank from one vendor keeps the seam simple.
If your constraints differ. Everything is behind a factory seam (make_embedder, make_reranker), so switching is a config change, not a rewrite. Want fully local and free? Point it at a sentence-transformers embedder and a local cross-encoder. Already on OpenAI? Swap in text-embedding-3-large. The project started on Voyage and moved to Cohere, that migration was a manifest and adapter change, which is the whole point of the seam.
The LLM, Groq, Llama 3.3 70B
Chosen: Llama 3.3 70B served by Groq.
For a streaming conversational assistant, tokens-per-second is a feature, not a nicety, a shopper watching a reply appear word by word forgives a lot. Groq’s inference speed makes a 70B model feel instant, and Llama 3.3 70B is more than capable of the grounded, cited, format-following answers this needs. It’s also open-weight, so there’s no lock-in to the model itself.
If your constraints differ. The LLM sits behind one make_llm seam and a .generate() interface. For higher reasoning quality on hard multi-step questions, point it at a larger Groq-served model first (staying on one vendor), or at a hosted frontier model if you accept the added dependency, the prompts are model-agnostic. For a stronger judge in the RAGAS eval, set JUDGE_MODEL independently so the app model isn’t grading itself.
The frontier model, measured and declined
Chosen: Groq only, no separate frontier vendor, decided on numbers.
The obvious “make it smarter” move is to add a frontier model for routing or for the escalation persona. I made that a measurement, not a preference. On the labeled routing set a 70B tie-break scored 85.6% against an 8B’s 85.9%, so a model roughly twenty times the price per token did not even edge out the cheap one on classification. The escalation persona’s job, gather the shopper’s details, confirm them, and file a ready case, is structured and format-driven rather than reasoning-heavy, which the 70B handles well. So the system stays Groq-only, and a frontier model is reserved for the one place it earns its cost: an offline, occasional judge on the hardest reviewed cases, never in the serving path. The numbers are on the evaluation page and the dollars are in the cost model.
If your constraints differ. The tier is a config value and the eval is one command, so revisiting this is run_agent_eval.py with a new model in the comparison. The discipline is to let the number, not the instinct, decide.
A knowledge graph, worth it, but scoped
Chosen: Neo4j, used for relationship questions only.
Vectors are bad at relationships. “Which supplier makes the Cloud Hoodie” is a graph hop, not a similarity search, and forcing it through vector retrieval gives vague, wrong-shaped answers. The graph earns its place for exactly that class of question, and it’s treated as authoritative only when the query names the entity, so it grounds rather than guesses.
Honesty about scope: of the two relationships modeled, one (supplier → product) is fully populated and genuinely useful, and one (product → store, derived from a handful of historical sales) is thin and mostly returns nothing. That’s documented on the Reliability page rather than hidden. The graph is a real capability, not a checkbox, which means being clear about where it’s strong and where it isn’t.
A governed metric layer instead of “let the model do math”
Chosen: A validated semantic layer (metrics.yaml) over read-only DuckDB.
Language models are unreliable at arithmetic and worse at making up plausible numbers. Any question with a number in the answer, return rate by size, the cheapest hoodie, per-size stock, routes to a governed metric: the model fills declared slots, a resolver runs a parameterized SQL template on a read-only connection, and the result comes back as a labeled evidence block that is never fused with vector scores. The model presents the number; it never invents it. Small samples even come back with the sample size attached, so a rate over six sales is stated as “based on only six sales.”
DuckDB + dbt for the lakehouse
Chosen: a DuckDB medallion (bronze → silver → gold) built and tested by dbt.
This is a portfolio, so the data layer has to demonstrate governance, not just have it. dbt gives real schema tests, PII-masking rules, and a deterministic, reproducible build, and it runs in CI, where a parity test asserts the dbt-built gold tables match the in-app builder exactly. DuckDB keeps the whole thing local and zero-infrastructure: no warehouse to provision, and the same file works on a laptop and in CI.
Three brains, one seam
Chosen: a master orchestrator (the default), a fast linear pipeline, and a LangGraph supervisor, selectable by one CHAT_BRAIN flag.
The default omni brain is the master orchestrator: it routes each turn to a specialized lane, then streams through the one gated pipeline. linear is that same pipeline with no routing, one pass, retrieve, ground, answer. The LangGraph agent brain is a bounded loop: it reformulates and re-retrieves when the first pass is thin, and escalates to a human review queue when it still can’t ground an answer. Same retrieval, same guards, different control flow. Keeping all three behind one seam means the demo can show routing and the agentic loop without slowing down the common case.
Alternatives at a glance
Every component is behind a factory seam, so each row below is a config change, not a rewrite.
| Component | Chosen | Alternative | Switch when |
|---|---|---|---|
| Embeddings | Cohere embed-v4.0 |
sentence-transformers (local, free) or OpenAI text-embedding-3-large |
You want zero quota (local) or you are already on OpenAI |
| Reranker | Cohere rerank-v3.5 |
a local cross-encoder (bge-reranker) | You need fully offline, or lower cost at higher volume |
| LLM | Groq Llama 3.3 70B | Claude or a GPT-4-class model | You need stronger multi-step reasoning over streaming speed |
| Vectors | Qdrant hybrid | pgvector, Weaviate, Milvus | You want vectors in your existing Postgres, or a managed cloud |
| Lakehouse | DuckDB + dbt | Snowflake, BigQuery, or Databricks + dbt | Data outgrows a laptop, or a team needs a shared warehouse |
| Graph | Neo4j | Memgraph, or a relational adjacency model | You already run one, or the graph stays small enough for SQL |
How it changes as the stakes change
The architecture was chosen so the answer to “what if it grows up” is an increment, not a redesign.
- If the domain becomes regulated (legal, health, finance). The manifest-driven masking test, the read-only governed queries, and the deterministic privacy guards (the order-PII gate that decides who may see an order) are already the spine of a compliance story. The additions are an audit log on every disclosure, a data-retention policy on the trace store, and a human-review requirement wired into the existing escalation queue. Nothing needs to be torn out to add them.
- If it goes B2B and multi-tenant. A domain is already a config folder, so a tenant is a pack plus an isolated index and graph. Auth moves from the demo gate to per-tenant identity, order access binds to the authenticated principal instead of a typed name and email, and rate limits and cost ceilings become per-tenant. The seams are the same.
- If the data moves to a warehouse or lake (Databricks, Snowflake, BigQuery). The dbt project is portable: point it at the warehouse and the same medallion, tests, and exposures run there. DuckDB stays as the fast local build for CI and development, so the reproducible offline story survives the move to a shared platform.
- If traffic scales. The in-memory rate limiter swaps for Redis behind the same interface, the vector store and graph move to their managed cloud tiers, the LLM and embedder are already concurrent-safe behind retrying seams, and the weekly continuous-training workflow gains shadow and canary stages on top of the drift check and eval gate it already runs. The instance cap that protects cost today becomes an autoscaling policy.