Who this is for, what their problem is, and what the product does about it. The market argument — size, buyers, competitors, expansion — is mrd.html. How anything is built is tdd.html. Why a choice was made over its alternatives is decisions.html.
Verbatim is regulatory change intelligence for one person: the regulatory affairs analyst at a multi-state investor-owned utility, the one who reads a new docket filing and has to work out, fast, what changed, whether it is binding yet, and what the business must now do.
Today that work is manual. A new draft or order lands; the analyst opens it beside the last version, finds the delta by eye, judges materiality against memory of the obligations register, and routes it by hallway conversation or email. An alert only tells them something moved. Search only helps if they already know what to look for. Neither one does the interpretation, and the interpretation is the expensive part.
Verbatim takes two versions of a proceeding and diffs them deterministically. A model is never asked to find the change; it is only ever asked to say something about a change it has been shown. Every claim carries a citation to an exact source passage, and that citation is re-read against the stored text before the claim is shown as fact. An unverified claim does not assert itself; it goes to review, labelled unverified, not hidden.
The hard part is exactly that: language models misquote fluently and with confidence, and in this domain a wrong citation is worse than no answer. Verbatim assumes the model will get it wrong sometimes and is built so that failure is visible and survivable.
The demonstration runs on a real filing: Kentucky PSC docket 2025-00113, a witness's direct testimony filed and then corrected by the filer — 1,024,409 characters against 1,024,536, a 127-character difference across a million. Segmenting both gives 4,285 passages against 4,287; the diff over them takes 0.43 seconds on this machine and reports 144 changes — 134 modified, 6 added, 4 removed. The filer also published their own marked-up copy, so there is an answer key nobody on this side wrote. make seed loads that filing. A seeded database then holds 20 proceedings, 105 versions, 889 changes and 107,257 passages. Recount rather than quote: sqlite3 verbatim.db "select count(*) from changes" answers in a second, and these figures move whenever the seed does.
One person, per ADR-01: the regulatory affairs analyst at a multi-state investor-owned utility. Not the compliance officer, who owns the obligation register but consumes analysis rather than producing it. Not outside counsel, who bills by the hour for exactly this work and has no incentive to make it faster. Not "regulated enterprises" in general, which describes an industry, not a person.
The analyst is the one who reads the docket. Their job is to turn a filing into a judgement — what changed, whether it matters, whether it is final — and to hand that judgement to the people who act on it. That is the job Verbatim is built to make faster and more defensible, not the job of filing comments or writing legal opinions.
data/real/ — 102 filings from eight commissions, filed into live proceedings by utilities, intervenors and commission staff, each record naming its filer, docket and date. No interview has been completed. The filer is an organisation on every record and only 19 of the 102 name a witness, so the corpus shows the work of this job without often naming the person who did it. A filing shows what somebody produced; it cannot show what it cost them, how often, or what they nearly missed. H1 and H2 under "User feedback" test the two load-bearing claims here directly.The analyst tracks several open proceedings at once, across several state commissions, because the utility operates in more than one jurisdiction. Proceedings move slowly and then suddenly — months of quiet, then a draft order posts, then a short comment window, then a final order with a compliance deadline attached. The analyst's job inside that rhythm is to notice the version that matters, work out what it changed from the last one, and decide, this week, whether the business has to do something and by when.
The moment Verbatim is built for: a new version of a proceeding the utility is a party to appears. The analyst does not want a summary of the new document — they have read the old one and know it well. They want the delta: what changed, in the source's own words, whether it is a draft they should watch or a final order they must comply with, which of the utility's existing obligations or projects it touches, and who on their team needs to see it this week. Today that delta is assembled by hand: two documents open side by side, a redline built by eye, a judgement of materiality made from memory, and a routing decision made by knowing who owns what. Verbatim's job is to produce that same delta as a verified, citable artefact instead of a manual reconstruction.
Both exist today and neither does the analyst's actual work.
An alert says something moved. A docket-tracking subscription or a law firm client memo tells the analyst a new filing posted. It does not tell them what changed inside it, whether the change is material, whether it is still a draft that could be withdrawn on rehearing, or which of the utility's obligations it touches. The analyst still has to open both versions and do the entire interpretive job the alert implied was done. An alert converts silence into a to-do item; it does not shrink the to-do item.
Search is worse, because it assumes the question. Search answers "where does this proceeding mention interconnection timelines" — a query the analyst can only write if they already know a change touched interconnection timelines. The analyst's actual starting point is the opposite: they do not know what changed yet, that is the thing they are trying to find out. A tool that requires the query to already contain the answer is not built for the moment it claims to serve.
What both tools skip is the same step: turning a new version into a judged, cited, routed piece of work. That step is the product. It is why the unit of work in Verbatim (ADR-02) is a change, carried through to a recommended action with a named reviewer — not a document, not a query result, not a notification.
An assistant did ship, and it does not contradict the paragraph above. Clarke reads the workspace record and only the record. Per ADR-50 it is a commission's records clerk rather than a guide: ask whether a change is material and it says that is the analyst's call, then offers what it holds — the versions, the diff, the obligation, the owner. Asked for something not on the record, it says so without embarrassment. A tool that can withhold has to return its withheld count, and a missing count stops the turn rather than letting it report zero (ADR-52). None of that makes search the wedge; it makes the record reachable by a person who would rather type a sentence than learn a screen.
A second assistant shipped, and it is not for the analyst. Say so rather than counting it as product. /explain draws the path a pair of filings takes and answers questions about it from this project's own PRD, MRD and TDD (ADR-95). It exists because a reviewer asks "why does that box exist" and the drawing cannot answer; the analyst in ADR-01 has no use for it. It obeys the same rule as everything else here: each answer's citations are re-read against the file on disk before the answer is shown, and one that does not re-read is shown marked unverified rather than quietly dropped. With no OPENAI_API_KEY the sections are ranked by bm25 instead of by embedding and every reply says so. The honest framing at a panel: this is a feature built for the person reading the submission, and by the rule that every feature maps to a named user pain it counts against the reviewer, not against the analyst.
One workflow, deliberately narrow, run against two corpora that do different jobs. The synthetic proceeding — three versions, known material and immaterial edits, plus a company context of eight obligations — carries traps built on purpose and is what the evals measure against, because a real filing will not reproduce a trap on demand. The real corpus in data/real/ is the demonstration: filings somebody actually made, with the commission's own URL beside each. ADR-40 refuses to let either corpus stand in for the other.
verify_citation_for_version, says in its own docstring that nothing in app/ calls it, so that check does not yet run in production.app/diff/engine.py — no model, no clock, no randomness. The model is not asked to find changes (ADR-04): diffing has a known right answer per test fixture, and asking a model to do it too would make the step untestable. Alignment confidence is capped when the section label disagrees, so a renumbering escalates instead of presenting itself as a confident match.
ANTHROPIC_API_KEY nothing is judged and the run carries a named off state with an announcement for the reader; it does not fall back to seeded text and present it as interpretation.app/ would let them (ADR-24).sqlite3 verbatim.db "select mapped_by_kind, count(*) from change_obligations group by 1" answers in a second. What is deferred is the semantic half: OBL-001 says "post security" where the docket says "post collateral", and no lexical rule reaches that. ADR-08 reserves it for embeddings; ADR-85 records what shipped instead and why the shape — propose, show the words, let a person confirm — survives the day embeddings land.ROUTE_MAPPING_UNCONFIRMED and returns the person it would have named as a candidate rather than as the owner (ADR-87). On the seeded corpus the product therefore declines to route 67 of the 69 mapped changes, and every unmapped change besides. That is the correct answer and it is a quieter demonstration: the reviewer sees a product refusing to assign rather than a product assigning. The refusal has a cure on the same screen — an analyst confirms or rejects a proposed mapping in one click, and the audit row records which of the two decided.
/actions renders the monitor, comment and comply vocabulary on a row: an analyst holding action.propose proposes one action on a claim whose citation verifies, and a person holding action.approve approves or rejects it. That screen is the only caller of app/auth/policy.py::can_approve, so segregation of duties is enforced on the request path rather than in a library beside it — approve your own proposal and the product answers 403 with the audit row that proves you wrote it. Two limits stand. Approving records the decision and performs nothing: it files nothing, tells nobody and moves no project state. And no effective date is stored on anything — the code has a predicate saying a final order needs one, and no column holds it./s/<token>, readable with no account at all. Five controls stand where the session guard stood, each a column rather than a convention: a token of 32 random bytes stored only as a SHA-256 digest, so a read of the table hands nobody a working link; one artifact, never a project, never a proceeding, never a list; an expiry, seven days by default and thirty at most, never absent; revocation at any time; and an audit row for every mint, every open with its address and verdict, and every withdrawal. Opening a link re-reads the citation against the stored source during that request, so a source edited after the link went out takes the statement off the page on the very next open — and the person watching a claim withdraw itself is outside the company.
app/ calls create_share_link(). The reader at /s/<token> is wired and public, the register at /admin/shares lists every link with its opens and stops any of them in one click, and no screen creates one — so a seeded database holds no link to open, and select count(*) from share_links returns 0. Revocation has the same shape: the module lets a sharer withdraw their own link, and the only screen that offers the act belongs to an administrator.The end state the analyst sees today: a list of changes, each with its citation and its draft-or-final status; a named reason wherever routing declines to name a reviewer — which, on the seeded corpus, is all but twice; and, for any claim whose quote does not match its source, a review item instead of a silent guess. A materiality judgement is there where somebody has opened the change and a key is set, and it is withheld again the moment its citation stops re-reading. What the list still does not carry is an effective date on anything, and an approval that performs work rather than recording a decision.
The complaint about the step after this one is that an item disappears. The analyst hands it off and cannot say who has it, how long that person has had it, what happens if nobody acts, or what comes next — so they chase it by email, which is the workflow this product claims to replace. ADR-48 answers with two screens: one that reads the route and one that draws it.
GET /workflow is the read-only half, open to anyone signed in, because the people who do not draw the route are the ones who have to be able to read it. It answers "where is my item and what happens next" as a numbered list of steps in the order an item meets them, each written as a sentence: a timeout rule becomes "after 24 hours with no answer, this goes to anyone holding the Regulatory Affairs role". The page carries no JavaScript at all — the one screen every signed-in person depends on must not be breakable by a script failing to load — and it does not soften a bypass: "after 48 hours this step is skipped and the item moves on without approval", with a step already skipped on a live item drawn at the same weight as an approved one and said to have been skipped rather than signed. That is the citation rule moved onto routing. A route that draws a bypassed step to look like an approved one is the same defect as a claim asserted on a citation that did not verify.
The /admin/workflows paths are the other half, behind the workflow.manage permission. A save refuses a timeout behaviour the product's vocabulary does not have. Activation refuses a route with two entry points, a loop, or a step nobody can reach, and refuses an assignee rule or an escalation target naming a role the company does not have. Activation checks that a role exists, not that anybody holds it — freezing this morning's staffing into a route would be the worse error — so a role nobody holds passes here and is refused later, at the moment an item arrives. Absence is denial applies to routing exactly as it applies to citations, but it applies at a different moment, and a reader should know which.
What is thin: a run has to be started by hand, and on a seeded database none has been. One screen starts a run — the escalation queue posts to /escalations/<id>/route, which enters that escalation onto the live route or says in a sentence why it did not. Nothing else starts one. Nothing advances one on a clock either unless somebody turns the job loop on: VERBATIM_JOBS_ENABLED is off by default, make run never starts the loop, and the deploy container starts it only when that variable reads true. A seeded database holds an active route, four escalations and zero runs, so the "where has my item reached" half says no run has been started rather than rendering an empty list that reads as "nothing has happened".
This wedge needs two versions of a proceeding before it produces a change. The first version of any proceeding Verbatim sees has nothing to diff against, and nothing fills that gap. Running extraction and citation verification on a single version standalone — so the analyst gets a verified reading of the document before a second version exists — is the plan, not the build: no branch for it exists in the TDD, the architecture, the data model, or the task board. Cold start is a real weakness of a change-centric design, and it is planned rather than designed.
It may be worse than a weakness. Rehearsing the interview against a synthetic persona produced one question nobody had thought to ask: of the last ten things that landed on you, how many had an earlier version you had already read? If the answer is two, the change-diff wedge is a minority case and cold start is not a gap in the product — it is the product, and ADR-02 is aimed at the wrong moment. It is the fourth of the eight questions that rehearsal left standing, in docs/synthetic-interview.html. Nobody said it, and it counts as evidence for nothing until a real analyst answers it.
data/real/ arrived as text somebody had already pulled out, and the product stores one .txt per filing beside a provenance record. The layer that would break first on a new source type is the layer this build has not written for any source type, including its own. What the corpus does buy is a harder test of the layer underneath: the text is dirty in the ways real filings are dirty — docket captions, line-numbered testimony, certificates of service, words hyphenated across a line break, and, in 9 of the 102 files, a soft hyphen or a ligature. Normalisation is exercised against real damage rather than against a corpus built to suit it.scripts/seed_roles.py, each named for a function rather than a rank, and no template holds both action.propose and action.approve. Nothing runs that script automatically — neither make seed nor the deploy entrypoint calls it — so a seeded database carries three roles, not ten. The roles are who may approve, not a second user the product is designed for.data/real/ holds filings from eight commissions, so the cross-jurisdiction data is here and the work has not begun. Nothing in the code knows that "final order" arrives on a different clock in Ohio than in Utah.An expert user does not trust a compliance tool because it looks confident. The metrics below are what this build proposes to measure trust by. Three of the six are results and three are still definitions. The harness runs in under a second, calls no model and opens no socket; it prints 5 of 5 metrics pass over citation verification, deliberate-corruption rejection, diff completeness, occurrence disambiguation and draft-versus-final routing. Three of those five answer a row below; the table is not a mirror of the harness. (make eval wraps it in an install and a seed, so the target itself touches the database.)
| Metric | What it measures | Status |
|---|---|---|
| Citation verification | Whether cited offsets are confirmed against the stored source rather than routed to review. The direct proxy for "does the tool ever assert something it cannot back up." | Measured. 20 of 20 recorded offsets verify — a count, not a rate, and the harness refuses to turn it into one. |
| Deliberate-corruption rejection | Whether the verifier refuses two corruptions built to slip past it. | Measured. 2 of 2 probes rejected: one a fabricated quote sitting at real offsets, one the boilerplate sentence quoted correctly with no occurrence stated. The second is the interesting one, because exact text at exact offsets is what a naive verifier would wave through. |
| Change detection against labels | Whether the deterministic diff finds every change the corpus was built to contain, and escalates the restructure it was built to trip on. | Measured, on five changes. 5 of 5 found, with the restructure escalating and the ordinary edits not. At n = 5 a single miss moves any rate twenty points, which is why it is reported as a count. |
| Obligation-extraction precision and recall | Against a hand-labelled eval set. This is the standard the "AI excellence" evaluators expect — a number you can defend, not an adjective. | Not reportable, and not for want of building it. A hand-labelled set exists in data/manifest.json, but it labels changes, not obligations, and it carries positive labels only. With no negative labels, precision cannot be computed at all — not estimated badly, computed. The harness says so instead of guessing, and fixing it needs either negative labels or real reviewer rejections. |
| Escalation-queue clearance | Whether the review queue stays small enough that clearing it is routine rather than becoming its own backlog. A queue nobody empties is a failure mode the peer project already lived through (ADR-06). | Open. The mechanism exists — an escalation carries an assignee and the moment it landed — but the only queue is the seed's, and no reviewer has worked one. This needs live usage, not more code. |
| Behavioural trust proxy | Whether an analyst, shown a verified claim with its source span open, stops re-opening the original PDF to check it. Directly testable in a user session; not testable from the build alone. | Open. Proposed instrumentation only. Depends on the user interviews and any hands-on session that follows them. |
The caveat the scorecard prints itself still stands: no model runs in make eval, so a green board says nothing about model behaviour. It is a regression gate on the deterministic spine, and a strict one.
The model path is scored separately. make model-eval-send builds a golden set over the one model call on the analyst's path — the materiality judgement — and scores it. It keeps two things apart that an averaged score would hide. Fabrication — an assertion whose citation the harness could not re-read — is measured twice: how often the model produced one, which is a property of the model, and whether any survived to an assertion, which is a property of the gate and must be zero. A miss — a verdict the gate withheld — costs an analyst an afternoon and blocks nothing, so it is counted apart with a different threshold. A model that produced ten unverifiable citations and a gate that let none through is the design working, and one number covering both would have said the opposite.
What it does not settle. The harness spends money, so it runs on a deliberate command and not in CI; CI runs it with nothing sent and requires the exit code that means the model was not evaluated, rather than letting a step imply coverage it has not got. Three cases carry the hardest comparison — cited correctly, reasoned wrongly — and three is not a sample. So the honest sentence is narrower than "the AI is measured": the model path has a scorecard with the right shape and a corpus too small to defend a rate off.
docs/.ai/tasks.html is closed as failed rather than left open: 45 messages went out over two days, four bounced, and nobody replied. The tracking table in docs/user-research.html is still three empty rows. What follows is the hypotheses under test and what would falsify each.What was built instead, and the line it must not cross. A synthetic persona was assembled from the 102 real filings — docs/synthetic-user.html — and an interview was rehearsed against it in docs/synthetic-interview.html. Both pages open by saying nobody said any of it, that it is not evidence, and that no line may be moved into this document as a finding. A convincing fake transcript is the one artefact that could do this submission real damage. The persona generates questions to go and check, never answers to rely on.
It earned its place by what it could not do. Asked to walk through last Tuesday, it had nothing — so that question survived to the real script, because a question a synthetic cannot answer is exactly the one worth a real person's twenty minutes. On H3 it was deliberately not asked anything: the persona was built by the same hand that built the product, so its agreement would be worthless and its disagreement would be theatre. The output was eight sharper questions, each asking for an artefact rather than an opinion — "open your sent mail, what was the last version comparison you sent anyone?" in place of "does anyone diff today?" On H4 it moved the needle slightly and settled nothing: the best evidence in the corpus that interpretation is expensive is an Ohio commission-staff witness who twice wrote that a provision differs from the other stipulation without saying how, and he is staff, not this user.
| # | Hypothesis | What would falsify it | What it decides |
|---|---|---|---|
| H1 | The analyst, not counsel, does the change-to-action work. | Two or more interviews say outside counsel does the actual reading and the analyst only files the result. | ADR-01 and the entire "one user" framing above. If this falls, the person in this PRD is wrong and the document gets rewritten, not patched. |
| H2 | Nobody reliably diffs versions today; it is done by reading, memory, and manual highlighting. | An interview shows a redlining tool they already trust and use routinely for this exact task. | Whether the deterministic-diff step (ADR-04) is displacing a real gap or a tool that already exists and works. |
| H3 | Trust requires seeing the source passage, not a confidence score. | An interview subject says a bare confidence score would be enough, or that they would not check the citation either way. | Whether citation verification (ADR-03) is the product's central bet or an engineering nicety. If H3 falls, the move, stated in advance: keep the verifier as a correctness mechanism, stop presenting it as the trust mechanism, and say so at the panel. |
| H4 | The expensive part of the analyst's work is interpretation and routing, not detecting that something changed. | An interview subject says detection is the actual bottleneck and interpretation is fast once they know where to look. | Whether the product is solving the right end of the problem. If detection turns out to be the bottleneck, the wedge should move upstream toward faster, more reliable change-finding rather than richer interpretation of changes already found. |
If only one interview lands before submission, that one conversation gets reported here honestly, with what it changed or why it did not. One real conversation reported straight beats three invented ones. If none land at all, that is reported too, and the hypotheses above stand as the design's stated bets rather than confirmed findings.
| Question | State |
|---|---|
| Which model, and what happens when it is wrong or unavailable | Closed by ADR-37. A model call happens only in a named module, never on the deterministic path and never where a citation is checked. Three modules call one today — app/interpretation/propose.py, app/chat/agent.py and app/explain/answerer.py — and each pins claude-opus-5 in code rather than reading it from the environment, so a silent swap leaves a trace. With no key there are named off states that announce themselves, not one that degrades quietly. ADR-37 says two modules; the code says three, and the ADR is behind it. |
| Where the confidence threshold sits, and on what evidence | Open, and blocked on the corpus. Settling it needs an eval set big enough to count the cost of a false escalation separately from the cost of a false assertion at several threshold values. The present corpus cannot do that, so the number stays at zero rather than being invented. |
| What the obligation-extraction eval set is, and who labels it | Open. Nobody has labelled one, and nobody here can. Ground truth labelled by the person who built the system measures agreement with oneself, which is a sanity check wearing the clothes of an accuracy claim. It needs somebody who does the job. |
| Whether the change-diff wedge is the majority case | Open, and it is the one that could move the most. See "Cold start" above. A single question to a real analyst answers it. |
None of these gets a placeholder answer. An invented number reads worse under questioning than an open item marked open.