Real metrics after launch
No number on these pages comes from real visitors yet. After launch, published metrics come from the metric sheet with their denominators and the date they were read, and until then every figure is a measurement on test sets written for the purpose.
A case study with metrics is only worth reading if the metrics are honest about where they came from. This page holds the rules every number on How it works follows, what will be published once there is real traffic, and what was measured before any visitor asked anything. It will change more than any other page here, because it is the one that waits for data.
The rules for a published number
- Every measured number says when and how it was measured. A measurement without a date and a method is not published.
- Every claim says how mature it is. Designed means decided and not yet built. Tested means a check that runs asserts it. Measured means it was observed, with a number, at least once. A claim moves up only with its evidence (deep-dive 15).
- Every rate comes with its denominator. Published metrics come from the same sheet I read, where each rate prints its numerator and denominator, and a zero denominator prints "no data" rather than 0% or 100% (ADR-0057).
- Nothing describes one person. No transcript, no example question taken from a real session, and no aggregate small enough to point at one visitor. Examples on these pages are written by me and are not taken from anyone's session (ADR-0055).
- What cannot be measured stays unmeasured. Three of the metrics I set out to track have no honest instrument, and they are not replaced by a proxy. Analytics and observability says which, and why.
What will be published
The candidates are the rows of the metric sheet that exist from the first phase: the share of answers that cite a source, how often the assistant had no answer and which topics it was asked about, the rate of invented citations caught by the filter, tool success, latency against the timeouts, how questions were refused, and cost per conversation and per model. Page counts are published as what they are, Cloudflare's sampled estimate of HTML requests, crawlers included.
Some numbers arrive later or not at all. The share of interactions resolved without a model needs a denominator that exists only once visitors can consent to analytics. Feedback on answers, and anything about how suggestions are used, arrive with the phases that build them. The full dashboard, and a case study with metrics and lessons learned, belong to the last phase.
Measured before launch
These were measured on sets of questions written for the purpose, not on visitors. Each links to the page that explains it.
| What | Result | When and how |
|---|---|---|
| In-scope questions refused by the scope gate | 0%, on a 133-case set and on a 30-case holdout written separately | 2026-09-24, Jev with the list of project and organisation names, refusing only on a majority (ADR-0081). See Model routing and cost control. |
| Prompt-injection attempts | 14 of 14 clean on every mechanical condition, for the model that answers here | One run per model, in deep-dive 11, September 2026. The three conditions that need judgement were graded by two judges from different companies, one of them from the answering model's own family (ADR-0062). See Guardrails, privacy and security. |
| Invented citations caught after generation | 1.9 per 1,000: one of 519 source ids did not exist | Two full runs of the test set in deep-dive 8, on an earlier answering model. The output filter catches exactly this, by set membership (ADR-0039). See Retrieval and context engineering. |
One run, a few hundred cases and questions I wrote myself are enough to find a broken design. They are not enough to promise how the system behaves with real visitors, and nothing on these pages claims that they are.
How far this has been proved
- Designed: every number on How it works says when and how it was measured, and the metrics published after launch come from the sheet with their denominators.
- Measured: the three results above, each once, before launch.