Analytics and observability
The owner can see what each question cost, which layer answered it, which tools failed and which topics had no answer, from one record per question made only of ids, fixed values and numbers. A printed metric sheet and four alerts do the rest.
Observability for an agent usually means logging everything and deciding later what to look at. Here the decision came first: every number the brief asks for was written as an expression over named fields, and only those fields exist. The result is a system I can debug and cost without ever reading what anybody asked. What that means for your privacy is on Guardrails, privacy and security. This page is about what the numbers are and what they cannot tell me.
One record per question
Each question produces one row, written once when the turn ends and never changed afterwards (ADR-0056). Every field is an id from the site's own list, a value from a closed set, or a number: which tier the turn reached, what the scope gate decided, what the question was about in one of nine words (ADR-0087), which rung of the fallback ladder served it, how it ended, which tools ran and how long each took, tokens in and out and how many were cached, the cost as the provider reported it, the latency, the sections cited, the citations the filter dropped, and the subject the assistant looked for when it had no answer. A write carrying anything outside its vocabulary is rejected, and the rejection counts as a defect.
One rule decides every duration. A duration that measures the service is a number. A duration that measures the person is a bucket or is not collected. Time to first token is a number. The time between your questions is one of five values, from "first" to "over an hour", because the cache decision needs only to know how many gaps fall between 5 and 60 minutes, and a finer clock would be a record of how long you thought.
Not in the record: the message, the conversation, the selected text, the answer, the refusal, a user-agent string, a referrer, any model reasoning, and any key that could join the row to memory.
A metric sheet instead of a dashboard
Every metric in the brief is an expression over those fields with its denominator stated (ADR-0057). A command prints them as a sheet, and each rate carries its numerator and denominator, so a rate computed on four turns visibly is one. A zero denominator prints "no data", never 0% or 100%. From the first phase the sheet has the share of answers that cite something, the rate of unknown questions and of answers cut short, the rate of invented citations, tool success, latency against the timeouts, the most asked-about topics, how refusals happen, cost per conversation and per model, and cache behaviour.
Three of the brief's metrics are declared unmeasurable rather than proxied:
- Time needed to find information. A clock on a visitor is the measurement the bucket rule forbids. The sheet reports a different, differently named number instead: questions per conversation before the first answer that cites a page.
- Share of visitors who understand the positioning. That needs asking people. Inferring it from scrolling would be surveillance.
- Opportunities attributable to the site. Attribution ends at a message in my mailbox. What happens after is outside every system here.
One more has a numerator and no denominator. The share of interactions resolved without a model is the number the whole routing design aims at. But a followed link or a page read makes no request to the agent, so in the first phase there is nothing to divide by. It becomes computable only once visitors consent to analytics, which makes it the one headline metric that depends on someone saying yes.
There is no dashboard before the last phase, because a page with one reader is a surface to secure and host for a presentation improvement. What pushes is four alerts to one mailbox (ADR-0052): monthly spend passing 50% and 80%, the daily or provider cap being hit, rate limits refusing more than 5% of turns in a day, and the cache reading nothing for 20 turns in a row. An alert carries counters and names, never visitor text.
Counting pages without a script
Content pages carry no tracking script in the first phase, only a 2 KB script that animates the layout and sends nothing, so the site cannot count readers the usual way. Once a day it reads the previous day's requests per page from Cloudflare's own zone analytics and keeps one total per route and day, with no address attached (ADR-0086). The number is Cloudflare's sampled estimate of HTML requests. It includes crawlers, since the free plan has no bot score. So it is labelled "HTML requests (Cloudflare estimate)" and never "visitors" or "page views". It is also not the missing denominator above: a request is not an interaction.
Nothing on a content page reports what you do there. The browser-side event vocabulary has a consent tier on every event, and in the first phase no event leaves the browser at all (ADR-0037, ADR-0053).
Judging the answers
The quality of answers is checked against a test set written for the purpose, not collected from visitors, because collecting it would mean storing what visitors typed (ADR-0058). It has 51 test questions, each with the sections its answer should cite or the outcome it should have, 14 prompt-injection attempts, two held-out sets, and 8 deliberately bad answers that the grader must catch. Every expected section is a locked id, so a content change that moves one fails loudly.
Grading has two layers (ADR-0059). Everything that set membership or a frame value can decide is asserted mechanically: citations resolve, destinations are real, no link or forbidden markup, the right tool was called. Only four things no assertion can reach go to a judge model: tone, depth, factual accuracy and honesty. The judge is a gate that fires below 3 out of 5, not a score anyone tracks. On 65 real answers it gave nothing below 3 and flagged nothing, so it was trusted only after it caught 8 bad answers out of 8. Two judges from different model families now grade, and each was measured slightly favouring its own family.
The set runs when one of four things changes: the instructions or tool definitions, the corpus, the model, or the generation settings (ADR-0060). A failed assertion blocks the deploy. A judge below 3 is reported to me, because a judge is an opinion and a rubric should not be a veto. When none of the four changed, results come from a cache and the run costs nothing.
From the next phase an answer can be rated with one of three fixed values. There is no text box, because a feedback field that accepts a sentence is a store of visitor text with a helpful face (ADR-0061).
How far this has been proved
- Measured: no store this system owns has a field that could hold visitor text. The database refused four kinds of bad write on the infrastructure probe.
- Tested: in the first phase, content pages carry no script beyond the 2 KB motion module, by the byte budget that fails the build and by a check against production after each deploy.
- Designed: page counts are Cloudflare's sampled estimate of HTML requests, crawlers included.
- Designed: three of the brief's metrics are not measured, and the share of interactions resolved without a model has no denominator in the first phase.
- Designed: the deploy gate on the test set, and the record, the sheet and the alerts. None has seen a real visitor yet.
- Measured before launch: the test set has been run against six models and graded by judges from two model families.