What Verbatim becomes past the 48-hour build, and what has to be true before each step is honest rather than aspirational.
The next four weeks exist to find out whether the mechanism survives contact with reality, not to add scope. Nothing in this section is a new feature; everything is the 48-hour build tested against something it did not choose. Items that moved during the build keep their place here with what they said and why it changed — a roadmap that quietly drops what it got wrong teaches nobody anything. The first item was finished and the second unblocked, both by writing code. The two that did not move are the two that need a person in the room.
Unlocks. ADR-003's normalisation rules were written against a corpus with no OCR artefacts, no re-paginated exhibits, no inconsistent quotation marks across a scanned PDF. Real filings have all three. Every rejection the verifier produces on real text is either a genuine catch or a normalisation gap — both are useful, and right now we cannot tell which is more common.
Needs first — no longer blocked. The precondition was a real proceeding in the workspace, and that has landed, so this is startable now rather than in four weeks. It has NOT been started. Three facts bound what may be claimed about the real corpus, each of them a thing somebody could otherwise get wrong: nothing in the repository runs verification over the quotes pulled from it for the blog, so that check has never happened; the verifier compares normalize(actual) against normalize(quoted_text), so the property is normalised equality and not byte equality; and the provenance records name PyMuPDF, pypdf and pdfminer, which read a PDF's text layer — so the corpus carries extraction artefacts, not OCR artefacts, and the two break a verifier in different places. What is unmeasured is everything: how often the verifier refuses real text because normalisation is short rather than because the quote is wrong. That needs a labelled sample of rejections and nobody has read one.
Where this actually stands. All four hypotheses in docs/user-research.html are open, and they are open on zero interviews, not on two or three. The interviews table has three empty rows and T1 on the board is closed as failed, not left open: 45 messages went out, four bounced, and zero came back. Cold outbound has been tried and it did not work, so "send more requests" is not the plan for the next four weeks — a warm introduction is. Four weeks of a working group — five to eight analysts, not a one-time conversation — is enough to actually falsify or confirm them, especially H3 (does trust require the source span, or would a confidence score do). H3's answer decides whether the depth area is a trust mechanism or only a correctness one.
What has moved, and what it is worth. Not the hypotheses. A synthetic persona built from the 102 real filings now exists — docs/synthetic-user.html and docs/synthetic-interview.html — and both pages say on their own first line that nothing in them is evidence and no sentence may be lifted into this document as a finding. What the rehearsal did produce is worth keeping: eight sharper questions, each asking for an artefact instead of an opinion. "Open your sent mail — what was the last version comparison you sent anyone?" settles H2 better than asking whether anyone diffs. That saves a real twenty minutes; it does not substitute for one.
Needs first. Analysts willing to use the prototype more than once. A single interview establishes interest; repeat use establishes whether the workflow survives a second Tuesday.
Unlocks. ADR-006 leaves the threshold as configuration with no evidence behind the number. It is worse than unevidenced: VERBATIM_MIN_CONFIDENCE_BP is documented in .env.example, ships unset, and falls back to zero, so it has never withheld anything. Tuning it is not adjusting a control, it is switching one on. Real reviewer decisions against real extractions are the only data that can set it honestly; a threshold picked before any reviewer has disagreed with the model is a guess wearing a number.
Half the precondition has landed. The queue can now be worked rather than only filled: an escalation carries an assignee and the time it landed on that desk, and a routing failure keeps the item visibly unrouted in the shared queue instead of losing it (ADR-036, ADR-056). Only the failure path can actually run, though — nothing writes the change-to-obligation mapping that routing walks, so on the seeded corpus every item refuses with "no obligation" rather than resolving to an owner. Note the one decision this tuning may not touch: a reviewer disputes a refusal, they never overturn it, so no threshold anyone picks can make an unverified citation assert itself (ADR-024).
Needs first. Reviewers who have actually disagreed. Amend is not built — it was cut, and the board says so — so today's queue records approve and reject only, which is a coarser signal than this item assumes. The working group above is the remaining gate.
What it is. A canonical map from each jurisdiction's own vocabulary — proceeding types, filing statuses, the exact trigger for "final" — onto one internal model. A state PUC's forty proceeding types do not line up with FERC's, and "final order" does not mean the same thing, or arrive on the same clock, in every commission.
Why this is the moat and not the model. Any competent team with access to a frontier model can rebuild the citation verifier and the diff engine in a few weeks — that part of the stack is replicable by design, because ADR-004 and ADR-007 deliberately chose boring, well-understood mechanisms over anything proprietary. The taxonomy is not replicable the same way. It is accumulated, jurisdiction-by-jurisdiction domain knowledge, wrong the first time it is written and correct only after real proceedings expose where it breaks. A fast follower can copy the architecture from this repo's ADRs in an afternoon. They cannot copy three months of "this is what Ohio calls a Tier 3 rulemaking and how it maps to Texas's equivalent" without doing the same work.
The precondition has quietly been met, and the work has not started. This item used to need "real proceedings from more than one jurisdiction", on the argument that a taxonomy designed from one state's data is not a taxonomy, it is that state's data with a general-sounding name. data/real/ now holds filings from eight commissions — Georgia, Indiana, Kentucky, Missouri, North Carolina, Ohio, Utah, Virginia. So the blocker is gone and nothing has been built: there is no mapping table, no canonical vocabulary, no code that knows Ohio's filing statuses differ from Utah's. The gap between "we now have the data" and "we have done the work" is the whole item. A met precondition is not progress.
Still first on the cut list. The 48-hour cut list (docs/.ai/tasks.html) sacrifices this first, and it was never started. That was correct triage under a 48-hour clock, not a judgment that it does not matter — it is cut first under time pressure and rebuilt first once the core loop works, because it compounds and the citation pipeline, once built, mostly does not.
Unlocks. T17 in the 48-hour build produces eval numbers against one hand-labelled corpus — data/manifest.json, five labelled changes and nine boilerplate spans over the three synthetic versions. A snapshot. Turn every reviewer correction in production into a labelled example instead, and the eval set grows on its own from real correction traffic, without anyone hand-labelling anything after week one. A static eval set caps how much anyone can trust the extraction step, no matter how good the underlying model gets; a growing one, sourced from real disagreement, raises that ceiling every day the product is used. This is the second compounding asset, alongside the taxonomy, and it is the direct answer to "how does this get harder to copy over time" — a question the panel is likely to ask given how replicable the base architecture is.
The specific defect that makes this urgent. The manifest carries positive labels only — the changes that should be found — and no negative ones. So make eval can report recall as a count and cannot compute precision at all, which the harness says out loud rather than estimating. Reviewer corrections are the natural source of negative labels: a change the model flagged that a reviewer rejected is the example the corpus does not have. That makes this item the fix for a hole in today's numbers, not only a scaling story.
Needs first. Real usage volume, and a mechanism — not yet built — to route reviewer decisions back into the eval set rather than just into the audit trail. That mechanism is new engineering work, not a byproduct of the reviewer queue that already exists.
Unlocks. The 48-hour build ingests files a human hands it. Pulling directly from a jurisdiction's e-filing system on a schedule closes the gap named in the user-research script's opening question — how they hear about change today — which is answered by an alert, a colleague, or a law-firm memo, none of which the prototype touches.
The cost of the manual path is now measured rather than guessed. The 102 filings in data/real/ were retrieved by hand, commission by commission, and the provenance records show how little the eight resemble each other. Each has its own document handler — Georgia's FACTS detail page, North Carolina's STAR handler, Virginia's SCC DocketSearch, Missouri's EFIS chain — and, more awkwardly, its own identifier scheme, so the provenance files carry facts_document_id for one commission, scc_doc_id for another and ncuc_document_number for a third. There is no shared key to build against. That is the first real evidence about what per-jurisdiction integration costs, and it argues the estimate up rather than down. It is evidence about the work and not a start on it: nothing in the committed application fetches anything on a schedule.
Needs first. There is no universal docket API — each jurisdiction's filing system has to be integrated individually, which is exactly why this item cannot be scoped before the taxonomy work names which jurisdictions matter first. It also introduces a new failure mode the 48-hour build never faced: a scrape that returns stale or partial results and looks, from inside the product, like a proceeding with no new activity. That has to fail the same way ADR-006 already requires — a failed or partial ingest escalates and is shown as failed, never silently rendered as "nothing changed." Building the ingestion pipeline without building that escalation path first would quietly break the absence-is-denial rule this whole product is built on.
Unlocks. The 48-hour build got further toward this than the roadmap expected. There is a Review screen rather than a bare queue; an escalation carries an assignee and the moment it landed on that desk; a five-step approval route covers all three timeout behaviours with named escalation targets; ten role templates name approvers by function, and no template holds both action.propose and action.approve; the audit chain records who did it, from which session. One caveat that applies to the route and the roles alike: both are seeded by scripts — scripts/seed_route.py and scripts/seed_roles.py — that neither make seed nor the deploy entrypoint runs, so they are in the repository rather than in a fresh database. At six months, with enough continuous use, that becomes the place the record actually lives — every action, its approver, its citation trail — replacing the spreadsheet that docs/user-research.html's "where does it live now?" question is designed to surface. That question sits in the script because the honest answer, most of the time, is a spreadsheet nobody trusts and everybody keeps anyway.
What still stops it being a system of record. Three things, none cosmetic. No effective date is stored on anything, so "its effective date" above is a promise this page cannot yet cash. Amend was cut from the review queue, so a reviewer approves or rejects and cannot correct. And ADR-032 says the reversal column exists while the digest scheme that would defend it does not, so nothing may treat a reversal as evidence — which is exactly the property a record needs to have.
Needs first. This is where a second kind of reader — a compliance officer, reading rather than producing — first has a legitimate claim on the product. ADR-001 is explicit that the build is for one user, and that discipline does not lapse here: the compliance officer becomes a consumer of state the analyst already produced, not a second person the product is designed for. Building for both before either usage pattern is proven is the exact trap the challenge's prep guide warns about, just deferred six months rather than avoided.
Unlocks. Once there is more than one customer, a pattern learned at one utility — "FERC changes to interconnection-queue rules are usually material" — could sharpen materiality judgment at another. That is real signal sitting on the table the day a second customer exists. Why this is hard, specifically. The useful signal (which classes of regulatory change turn out to be material) and the sensitive payload (whose obligations, whose internal wording, whose projects) are generated by the same event and live in the same record. A regulated enterprise cannot let a vendor pool its obligation text or company context across customers — that is legal exposure for the customer, not a preference Verbatim can trade off for a better model. Any approach that fine-tunes on customer text or shares an embedding index across tenants leaks the sensitive part to get the useful part. Needs first. A design that separates the two before a single line of pooling code exists: learn only over public regulatory text and an anonymised material/immaterial verdict, computed per proceeding, and keep company-context matching — obligations, projects, documents — strictly per-tenant, never pooled, never fine-tuned on. That boundary needs to be written down, reviewed by someone other than the engineer who built it, and probably described in customer contracts, before it is trusted. It is an organisational precondition as much as a technical one, and it is not yet designed — this page states the shape of the constraint, not a solution to it.
The bet. If the chain from source passage, to verified citation, to obligation, to action, to reviewer approval stays intact and exportable over years of real use, it stops being a UI convenience and becomes the audit artefact itself — the thing an internal auditor, or a regulator's own investigator, would want handed to them directly instead of reconstructed from an analyst's inbox and a spreadsheet. That is a different sales motion than "better compliance software": the artefact could be produced for the auditor as readily as for the customer, because it was built to survive scrutiny from the start rather than assembled after the fact to satisfy it. Needs first. Three things, none of which the 48-hour build or the 12-week or six-month work can substitute for. First, years of continuous, unbroken production trail — an audit artefact is worth something because of what it has been through, not what it could theoretically do on day one. Second, outside validation that the citation-verification chain actually meets an auditor's or regulator's bar; an engineering team cannot self-certify that, and no amount of internal confidence changes who has to sign off on it. Third, the six-month system-of-record work has to hold under real reviewer load first — this cannot be the first thing that has to work. The least validated claim on this page. Nobody has been asked whether an auditor would actually want this artefact in this form. That question belongs in the next round of user interviews, not six months in.
A roadmap with no exclusions has no point of view. These are excluded on purpose, and the reason is the same for most of them: each one would dilute the wedge in ADR-002, add a persona the product was not built for, or spend effort on a part of the stack that is not where the moat is.
| Excluded | Why |
|---|---|
| Summarising a filing, or any prose the product cannot quote the sharpest exclusion here | A summary is a paraphrase, and a paraphrase cannot verify against source bytes. That is not an implementation gap to close later; it is a difference in kind. The moment the assistant emits a sentence that is not quotable, the product's central claim acquires an exception — and the exception would immediately be the most-used feature in it. Everything else here exists to prevent exactly that: the withheld claim, the occurrence check, and the rule in app/chat/persona.py that a withheld statement may not be repeated, paraphrased, summarised or hinted at. Whoever wrote that rule understood that a paraphrase is an assertion wearing a softer coat.
It is also the commercial line. Every document tool summarises. If a competent team could rebuild this in six weeks, "we refuse to summarise, we only quote" is an answer to why they would not want to; "we also summarise" is not. What replaces it is in the roadmap above: a coverage map that answers a question across filings with one verified span from each, and an explicit silence where a filing says nothing. A summariser would smooth those silences into prose and the reader would never learn they were there. |
| General regulatory search or a chatbot over arbitrary dockets line moved, not crossed | ADR-002 rejects search as the wedge: it assumes the analyst knows what to look for, and the expensive part — interpretation — is exactly what search skips. An assistant has since shipped, so this row has to say what was excluded and what was not. Clarke reads the workspace record and nothing else. ADR-050 makes it a commission's records clerk: it answers with the versions, the diff, the obligation and the owner, and says "nothing on the record for that" rather than obliging. Asked whether a change is material, it says that is the analyst's call. ADR-052 makes a tool that can withhold return its withheld count, and a missing count stops the turn rather than letting it claim zero. What stays excluded is the thing this row was always about: a chatbot answering over arbitrary dockets, off the record, with no citation to withhold. |
| Drafting comments, testimony, or filings on the analyst's behalf | Verbatim's job is reading and routing, not advocacy. A model asserting text into a docket is a materially higher-stakes failure than a model misreading one — that is counsel's liability profile, not this product's. |
| A fine-tuned or self-hosted foundation model, near term | ADR-004 places the depth in the deterministic pipeline around the model, not the model itself, and ADR-037 draws the boundary in code: exactly one module may call a model, the id is a pinned constant so a silent swap leaves a trace, and with no key the path names which of three off states it is in rather than degrading quietly. Training one is effort spent where the moat is not — the taxonomy and the eval set are the compounding assets; the model is a replaceable component by design. |
| Multi-tenant cloud infrastructure ahead of a second real customer | ADR-028 keeps SQLite as the default and supports Postgres through VERBATIM_DATABASE_URL, with a named trigger to move: a second tenant, or any real customer data, whichever comes first. It is close to a configuration change and not free — that ADR lists the guarantees which are application promises until the move, row-level security among them. Scaling infrastructure before there is load to scale for is waste against a clock that should be spent on the taxonomy and the evals instead. |
| A second named persona before the six-month system-of-record work | ADR-001's discipline: one user until that user's workflow is proven. The compliance officer's turn comes after, as a reader of state already produced, not before. |
| Any pooling of raw customer text across tenants | Not deferred — excluded outright until the separation design in the six-month section exists, is reviewed, and is contractually described. This is a hard line, not a backlog item waiting for capacity. |