Data

Where the data comes from, and how it becomes answers

An assistant is only as good as the data behind it. This page follows a single domain pack from raw files to the three things the assistant actually reads: a vector index, a governed metric layer, and a knowledge graph. The same code runs any pack, because the shape of the data is declared in a manifest, not compiled into the engine.

One pack, two kinds of data

A domain lives in one folder. Structured facts are CSVs, unstructured content is JSONL, and a single domain.yaml manifest declares the schema, the PII columns, the graph edges, the metrics, and the starter prompts.

flowchart LR
  subgraph Pack [domains/apparel_ecommerce/]
    direction TB
    S["seed/structured/*.csv<br/>products, orders, suppliers, stores, sales"]
    U["seed/unstructured/*.jsonl<br/>descriptions, reviews, guides, orders"]
    M["domain.yaml<br/>schema · PII · graph edges · metrics"]
  end
  S --> BRONZE
  U --> CHUNK
  M -. drives every stage .-> Pack
  subgraph Analytics [Structured path: DuckDB + dbt]
    direction TB
    BRONZE["bronze<br/>raw, lineage-tracked"] --> SILVER["silver<br/>typed + PII masking"] --> GOLD["gold<br/>curated marts"]
  end
  subgraph Semantic [What the assistant reads]
    direction TB
    METRIC["governed metrics<br/>read-only DuckDB"]
    GRAPH["knowledge graph<br/>Neo4j"]
    VEC["hybrid vector index<br/>Qdrant"]
  end
  GOLD --> METRIC
  GOLD --> GRAPH
  CHUNK["chunk + context prefix<br/>Cohere embed-v4"] --> VEC

  classDef pack fill:#e9f3f4,stroke:#0f7b84,color:#13191e;
  classDef gold fill:#f8f2e6,stroke:#c8791f,color:#13191e;
  class S,U,M pack;
  class GOLD gold;

The structured path: a real medallion

The structured files run through a dbt medallion, the same bronze to silver to gold you would build on a warehouse, but local in DuckDB so it needs no infrastructure and reproduces byte for byte.

  • Bronze keeps the raw rows verbatim, so lineage always traces back to the source.
  • Silver types every column and masks any column a pack marks as PII (the value becomes a masked: hash), enforced by a dbt is_masked test that fails the build if a raw value slips through to gold.
  • Gold is the curated set the assistant and the dashboards query.

Every build runs dbt tests: not_null, unique, relationships, and the masking test on any declared PII column. A parity test in CI asserts the dbt-built gold is identical to an in-app Python builder, so the two ways of producing gold can never quietly diverge. dbt exposures name the downstream consumers, so lineage answers “what does this table feed” for a reviewer, not just “what feeds this table.”

The demo’s structured tables (products, stores, suppliers, sales) carry no personal data, so nothing is masked here today; the customer PII lives in the order documents on the unstructured side, where a deterministic gate, not the model, decides who may see it (see Reliability). Point a pack with a PII column at the same medallion and the masking test starts guarding it, no engine change.

The unstructured path: chunk, ground, embed

Descriptions, reviews, and guides are chunked with a short context prefix (the product they belong to), embedded with Cohere embed-v4.0, and indexed in Qdrant as hybrid vectors, dense meaning plus a sparse term leg, so “a warm rain jacket” and an exact SKU both land.

The knowledge graph: how things relate

The graph is the part that answers “how are these connected.” Nodes and typed edges are built from the gold tables, never hand-authored, so the graph and the tables never disagree.

flowchart LR
  SUP[Supplier] -- SUPPLIES --> PROD[Product]
  PROD -- SOLD_AT --> STORE[Store]
  REV[Review] -- MENTIONS --> PROD
  DOC[ProductDoc] -- DESCRIBES --> PROD
  classDef auth fill:#e9f3f4,stroke:#0f7b84,color:#13191e;
  class SUP,PROD auth;

Zooming in on one product, “which supplier makes the Cloud Hoodie” is a single hop from Product to Supplier over the SUPPLIES edge, returned as authoritative grounding rather than a fuzzy guess. Traversals are allowlisted (no free-form Cypher), which keeps the graph injection-safe. The graph is treated as authoritative only when the query names the entity itself, so an entity merely mentioned in retrieved text enriches the answer without overriding it.

Honest scope. The SUPPLIES relationship is fully populated and genuinely useful. The SOLD_AT store relationship is derived from a handful of historical sales, so it is thin and returns nothing for most of the catalog. It is a modeled capability that needs real inventory data to be broadly useful, and it is called out as such rather than hidden. See Reliability.

Why a semantic layer

The one file that ties it all together is metrics.yaml. It defines each governed number (return rate by size, cheapest in a category, per-size stock) once, and the assistant, the evaluation harness, and the admin dashboards all read the same definition. A number is never free-form SQL and never invented by the model: the model fills declared slots, a resolver runs a parameterized single-SELECT on a read-only connection, and the result comes back as its own cited evidence. One definition, many consumers, is what “governed” means here.