Thirty-eight questions, each with the answer said out loud and the file the answer rests on. Every claim below was re-checked against the code on 2026-08-05 at commit a5c20cb, and the counts refreshed at 3ab55b7 — against the tree, not against the documents, which disagree with it in four places named at the foot of this page. The suite was re-run rather than remembered, most recently on 2026-08-08 at commit ba5965b through make fresh-check: 2,785 passed, 1 xfailed. It takes five to six minutes depending on the machine: 296–311 seconds here, 361 on an independent reviewer's run. Quote the range, not a point — the flattering end of a spread is still an overstatement.
If you rehearse nothing else, rehearse these. Between them they answer all four evaluation dimensions.
WarmupYour citation verification runs when a page is read, not when the claim is written. Why that way round, and what does it cost you?
Because a write-time verdict is a promise about bytes that can move afterwards. verified_claims re-reads the stored source at the cited offsets every time a claim renders, so editing the source flips the claim to withheld on the next page load — no job runs, nothing rebuilds, the page just refuses. It costs about nine milliseconds on a fifty-three kilobyte filing and a hundred and seventy on the million-character Kentucky testimony: linear in the document, not in the quote. The fix is to cache the projection and never the verdict — a cached verdict is the write-time promise again.
Evidence. app/state/claims.py:502 (verified_claims); app/verification/verifier.py:151-201. Measured on this machine this morning rather than guessed: occurrence_count 8.9 ms on va-scc-PUR-2026-00056-odec-dembek-corrected-testimony-clean-redline.txt (53,132 chars); verify_citation 168.7 ms on ky-2025-00113-kollen-direct-testimony-corrected-clean.txt (1,024,536 chars), of which occurrence_count alone is 169.5 ms. Nearly all of the cost is the ambiguity check, which projects the whole document; the string comparison itself is 40 microseconds. Linear scaling from 8.9 ms at 53 KB predicts about 180 ms at a million characters and the machine gives 169, so the shape holds. It is neither cheap nor constant, and the 345 ms once published does not reproduce.
CoreWhere does modern AI fail, and how did you design around it?
It fails by being fluently, plausibly, subtly wrong in a way nothing downstream can detect: a model reads a paragraph well and remembers where it read it badly. Concretely, it hands you a quotation that reads exactly like a commission order and is not in the document — no confidence dip, no hedge, nothing after it that can tell. So the design assumes misquotation and makes it survivable rather than trying to prompt it away — every claim carries version, offsets and quoted text, and the source is re-read before anything is shown. A claim that fails is withheld, not greyed out, and nothing is repaired: snap an offset to the nearest match and the gate is checking the repair.
Evidence. ADR-03 at docs/.ai/decisions.html:54-59; app/state/claims.py:302-322 — WithheldClaim is frozen with slots=True and has no statement field, so the template cannot render one by accident; app/web/templates/change.html:4-9; app/interpretation/propose.py:67-71 ("NOTHING IS REPAIRED"). With no API key the run carries MODEL_PATH_OFF_NO_API_KEY and announces it at propose.py:195 rather than quietly answering "not material": unjudged and judged-harmless look identical on a screen and mean opposite things.
WarmupWho is this for? One person.
The regulatory affairs analyst at a multi-state investor-owned utility — the person who opens the filing, works out what changed, and tells the business what it now has to do. Not counsel, not the compliance officer, not the executive: counsel bills by the hour for exactly that reading and has the opposite incentive, the compliance officer owns the obligation register but consumes the analysis rather than producing it, and "regulated enterprises" is an industry, not a person. That is ADR-01, written on day one with the alternatives beside it. It is a hypothesis rather than a finding, because nobody has confirmed it.
Evidence. docs/.ai/decisions.html:37-43 — ADR-01, alternatives at :40, the open hypothesis at :43. docs/prd.html:57. Every routing decision in the code assumes one reader and one approver, so the choice sits upstream of the unit of work, the routing walk and the screens: if two interviews say counsel does the real reading, the ADR gets rewritten and the PRD with it, and the ADR says so itself.
KillerDid you talk to real users before you built this? Did their feedback change anything?
No, and no. Zero interviews: forty-five messages went out to people named on certificates of service, four bounced, nobody replied. What I did instead was read a hundred and two real filings from eight commissions and build a persona where every trait points at a line in one of them, with the guesses tagged as guesses. That buys the shape of the job and what error looks like in it; it does not buy frequency, price, or what she would abandon.
Evidence. docs/user-research.html:45-50 (45 sent, 0 replies, 0 interviews, 4 bounces), :127-131 (the interview table is three empty rows), :137 ("This table has no rows"), :144 (the bounce, and what it changed in the verifier). data/real holds 102 .txt files across 8 commissions. The row about what changed because of user feedback says nothing, because nothing did.
HardWhat is the one thing you would rewrite?
The pairing between a claim and its source. It reads the whole tenant's corpus to build a map keyed on version id when it needs one row, and the drift check beside it — hash the version's text against what ingest recorded — is written, is tested, and nothing in the application calls it. So edit the cited span and the claim withholds on the next read; edit twenty characters elsewhere in the same filing and it still asserts against a document that is no longer the one we ingested. One scoped single-version read closes both halves.
Evidence. app/state/claims.py:469-499 — the docstring names the gap itself; app/verification/verifier.py:204-262, hash at :255-260. Checked with grep rather than trusting the docstring: grep -rn verify_citation_for_version app/ tests/ scripts/ shows every hit under app/ is a docstring, and the only callers are in tests/test_verification.py.
The read-path question and its cost live in the five. Start there, then these two.
CoreThere is no stored "verified" column. But a hash of the source would tell you the document has not changed. Why would that not do instead?
Because a hash answers a different question. A matching hash says the whole document is unchanged, not that the quote sits at those offsets, so you still have to read them; and a mismatch is too blunt the other way, because one unrelated typo would withhold every claim in that filing. So the hash is an addition, not a replacement. I do hash, in verify_citation_for_version, but nothing under app calls it, so drift is caught on cited spans and missed everywhere else in the same filing.
Evidence. app/state/claims.py:469-499 (the docstring names the gap rather than hiding it); app/verification/verifier.py:204-262, hash at :255-260. Only tests call it, and I checked with grep before saying so. Measured: sha256 of the 1,024,536-char file, 0.57 ms. It is a named hole, and the fix is the same edit that fixes the whole-corpus read — a scoped single-version read in queries.py.
KillerThe quote a claim cites appears twice in the filing. What happens?
The claim is refused. The verifier counts occurrences, and if there is more than one the citation has to state which one it rests on or nothing is asserted — the same sentence in two sections is two different obligations. The model's own offsets cannot settle it either: deriving the occurrence from them makes the gate check the model against itself. This is not a synthetic problem: in the Kentucky testimony we ingested, the phrase Cane Run BESS appears four times, and cited with no occurrence it refuses, as occurrence zero it verifies, as occurrence one it refuses.
Evidence. app/verification/verifier.py:137-148, 196-199; tests/test_occurrence.py; app/interpretation/propose.py:72-80. The synthetic corpus plants the same trap on purpose. Measured live: occurrence_count("Cane Run BESS") is 4; occurrence 0 verifies; no occurrence and occurrence 1 both refuse. Refusal throws away correct judgements, and the prompt asks for a unique span so it happens less often: rejecting a good citation costs a review, accepting a bad one costs the product its reason to exist.
"Where does modern AI fail" is in the five.
HardWhat is the model actually allowed to do in this product?
One thing: it is shown a single change the deterministic diff already found — both sides, the section, the exact offsets — and asked whether it matters to the utility, in one sentence, with a citation. It is never asked what changed, and never asked whether a version is draft or final. Its answer then passes five structural drops before the verifier is reached: malformed, a version this change does not span, an empty quote, offsets out of range, and a span outside the change it was shown. That last one matters most — a citation can verify perfectly and still be about text the model was never handed.
Evidence. app/interpretation/propose.py:82-88 (draft or final read from the field, ADR-05), :194-197 (the named fallbacks), :214-231 (the DROP_ codes including DROP_OUTSIDE_THE_CHANGE); ADR-04 and ADR-05 in decisions.html. The model is never asked what changed because the diff knows that exactly and can be tested against known answers; draft or final is read from a column and stated to it as fact. The prompt drifted from the ADR once and no test caught it, because every test asked what came out of the call; tests now assert what goes in, and tests/test_live_transport_probe.py:288 holds the request block to the exact bytes. This transport ran against the real API once deliberately, on 2026-08-05; every test drives an injected fake client.
KillerYou keep saying the live model path ran once. Is that true?
No. A second request almost certainly went out on the same day: an agent ran the probe script with send, believing the key was absent because it had cleared the environment variable, and load_env put it back. The output was piped to head and lost, so I cannot tell you the response, the token counts, or whether it succeeded. What I can defend is the one deliberate call: request bytes, response fields, stop reason, usage and cost all written down, and a test that holds the request block to the exact bytes.
Evidence. docs/.ai/live-transport-probe.html:34 — the correction note sits directly under the headline sentence and names the second call, so the page says this before an interviewer does. tests/test_live_transport_probe.py:288 rebuilds the request through AnthropicTransport and asserts it byte for byte. load_env puts back any name from the dot-env file that is not already set, which is exactly what clearing the variable creates.
KillerSo which parts of your model code have actually talked to Anthropic?
Two paths call the API. The citation gate in propose.py — ran once, recorded, on the fifth of August. The assistant in app/chat/agent.py — runs live on the deployed instance, but we never recorded a transcript. Everything else in the tests drives a fake SDK client. Line 322 is the half of that file I believe: a key reaches the assistant through config.py on the deployed instance, while the docstring at line 33 still says it never has.
Evidence. The assistant's transport carries a different parameter set. app/chat/agent.py:33 ("This path has never run against the real API in this repository") and app/chat/agent.py:322 ("The real thing, and it now runs") contradict each other inside one shipped file. docs/.ai/live-transport-probe.html:45 still lists the chat transport as not exercised. Both transport classes are driven by tests against a fake SDK client — tests/test_propose.py:659, tests/test_chat_agent.py:931 — so what is unexercised is the endpoint, not the class. Closing it takes the probe treatment for the chat transport and a recorded transcript.
HardYour diff is difflib. Where does it break?
On wholesale restructuring. Sequence alignment sees a section renumbered and relocated as a deletion plus an addition, and the dangerous part is that text similarity runs high exactly when structural identity has changed — our corpus carries the case on purpose, where section 6 becomes subsection 5.4 and raw similarity is 0.944. So a disagreement in the leading section number caps the score at 0.5. It is a ceiling, not a judgement: the alignment became an inference, and an inference escalates.
Evidence. app/diff/engine.py:1-11 (the weakness, disclosed), :22 (RESTRUCTURE_CONFIDENCE_CEILING = 0.5), :53-75 (_alignment_confidence); data/manifest.json CHG-5. Measured live: _similarity 0.9443, _alignment_confidence 0.5000, labels "6" and "5.4". The 0.5 ceiling sits below any plausible escalation threshold. Reported as a confident 0.944 match, a restructure looks like a small edit, which is how a moved obligation goes missing. Embeddings were the obvious alternative for the alignment and lose on the same ground: they find drift, not edits, and cannot name the words that changed, and the words are what the citation has to quote.
HardRetrieval. Why FTS5, and what happens when the index is wrong?
FTS5 is compiled into the SQLite the app already opens, so there is no service and nothing for a reviewer to install before the run command works. More important, the index points at passage rows that already carry version, ordinal and character offsets, so a hit is already in citation shape and composes straight into the verifier. When the index is missing, stale or unavailable the read does not return fewer results: it falls back to a full scan of every passage in scope and announces which path answered and why. The index and the verifier do not merely match on tokenisation: the rule is directional, the index may be more permissive and never less, because a restrictive index causes a silent miss nobody can see.
Evidence. app/state/search.py:27-40 (why FTS5, why Passage), :62-88 (the three named degradations and what the staleness check misses); app/state/queries.py:258-277 (the whole-corpus fallback). ADR-71 at decisions.html:689-696 records 366 ms scanning against 1.3 ms indexed on 8,707 passages. Most retrieval chunks text and then cannot cite what it retrieved. A short list with no announcement is the worst outcome available, which is why the fallback announces itself; and every hit is checked against its fingerprint. What the staleness check misses is a passage edited in place, indexed under its old words and never returned; it is inert today because nothing under app/ updates a Passage row. ADR-71's figures come from the fully seeded corpus, and that corpus has since grown: the 52 documents nothing had ever loaded were ingested on 2026-08-10 (gap P2-17), taking it from 8,707 index rows to 107,257. Re-measured there, indexed retrieval runs 8.36–10.62ms against 2,799–2,826ms scanning — 307 times. ADR-71's own numbers are kept as the dated reading they were. A partly-seeded working database holds far fewer passages than index rows, and retrieval on it degrades to a scan. verbatim.db is gitignored at .gitignore:5 and built by make seed on a reviewer's machine.
HardThe brief asks you to map a change to the documents it affects. Where is that?
It does not exist, and the schema says so out loud. An obligation carries a source document reference — the string DOC-2 — and the comment beside it says: not a foreign key, those documents have no table, they are entries in a JSON file. So there is no document to re-read, no offsets, no verification; the chain I actually have is change to obligation to a document label. Calling that document mapping would be exactly the fluent unverified assertion this product exists to refuse.
Evidence. app/state/models.py:1513-1518 (the comment and the column); app/seed.py seeds the four company documents as Source rows with locator data/company_context.json#DOC-n and no body text at all. data/company_context.json holds 4 documents and 8 obligations. Naming the column with an id suffix would promise a join nothing can satisfy. Closing the missing hop takes giving the tariff a table and ingesting it the way a docket version is ingested — passages, offsets, hash — after which the same verifier works on it unchanged.
KillerYour change-to-obligation mapping is a word overlap. That is barely AI. Why is it in the product at all?
Because the alternative was worse: nothing wrote that table, so every change in the product refused with no obligation mapped. It is deliberately not the semantic join — ADR-08 reserves that for embeddings I did not build. Four rules, each measured against the corpus: four characters or more; prefix match rather than substring; a word more than half the company's duties use is not evidence; and two words, not one. The part that matters is downstream — a mapping the pipeline proposed cannot put work on a person's desk. It is not good enough, and the limit is that "with", "each" and "before" are four characters and everywhere: a document-frequency weight is the fix, it is not built, and mapping.py says so itself.
Evidence. app/state/mapping.py:1-10 (the hole it fills), :12-27 (ADR-08 and note_for_semantic_join), :55-88 (the four rules, with "work" inside "network" — every network upgrade was proposing the security duty — and "large" in six of eight); app/state/search.py:259 (PREFIX_MIN = 4); app/state/routing.py:184 and :940-975 (ROUTE_MAPPING_UNCONFIRMED; the candidate goes in candidate_user_ids and never user_id). The company context is arranged to defeat anything less than a semantic join: one duty says post security where the docket says post collateral. Every candidate prints the words that produced it, so a candidate resting on "each" and "with" looks exactly as thin as it is, and rejecting one is a recorded decision rather than a dismissal.
CoreTenant isolation. Why a chokepoint rather than a code review rule?
Because a rule three people have to remember fails on the fourth call site, and it already did here. There are two guards: one refuses a company id that could match more than the caller meant — blank, whitespace, padded, or carrying a wildcard — and one refuses another tenant's row, and it exists because the first got mistaken for it. A function called the scope guard, passed because the string was well formed, then fetched by primary key and handed back another tenant's row. Now the fetch and the check are one call, and a foreign row raises rather than answering None, because None collapses nobody holds this into somebody else does.
Evidence. app/state/queries.py:1-23 (the two-guards docstring, naming routing.py::ensure_obligation as the failing site), :57 (SCOPE_WILDCARDS), :60-107 (_require_scope), :113-172 (row_for_company, CrossTenantRow at :168). Ids are unique across tenants and printed on the front of filings. The guard was restated instead of imported in one more place: the audit log wrote if not company_id: and verify_chain answered True over a scope it could not parse, which is the worst place in the product for that. It calls _require_scope first now.
CoreYour audit log carries three digest schemes. Why not migrate them to one?
Because migrating them would destroy the thing they exist to prove. Rows written before attribution existed still verify under scheme one, forever; re-hash them under the newest scheme and you get a chain that verifies and proves nothing, because the process that verified it is the process that rewrote it. So each row states its own scheme in a column and verification dispatches per row, and a chain mixing all three still verifies end to end. The migration adds columns and backfills the version number; it never touches a hashed field. Three separate digest functions are deliberate rather than debt: a shared helper is one edit away from moving every row at once.
Evidence. The three schemes: one hashes the ten original fields; two adds who acted, from where, under which session; three adds the reversal pointer. app/state/audit.py:13-24 (module docstring), :95-126 (the scheme comment, CURRENT_DIGEST_VERSION = DIGEST_V2 at :126), :262 / :309 / :369 (the three digest functions), :430-506 (_digest_for_row), :691 (verify_chain), :799-820 (migrate_audit_schema: "no rehash step and there must never be one", :804). A mixed chain verifies because the previous hash is a value copied forward and does not know which function made it.
KillerWhat is a reversal pointer, and what stops somebody writing one onto an old row?
Undo is a row, not an edit: taking back a decision appends an event naming the one it reverses, and that name is the pointer. Nothing has to stop somebody writing one onto an old row, because schemes one and two do not hash that column, so a pointer there is an assertion nothing defends and the verifier refuses the row outright. Scheme three does hash it, for the same reason attribution sits inside the hash: a column beside the hash can be re-pointed while the chain still verifies, and the log would then assert with a valid hash that a decision nobody undid had been undone. Application code cannot make such a row; one in that state arrived out of band, which is why a write-side check would not cover it.
Evidence. app/state/audit.py:26-32 (rollback is a row: the reverted row is not edited, not deleted, not flagged, because the record of a correction is exactly what an auditor comes for), :109-116 (the hole and how it is closed), :126 (CURRENT_DIGEST_VERSION stays at 2), :369-427 (_digest_v3), :430-453 (_digest_for_row raises AuditTamperError on a v1 or v2 row carrying a pointer), :216-217 (the two undo action codes, which differ in what they restore: decision.reverted restores state, decision.reversal_withdrawn restores nothing, and reading the second as the first puts a decision back in force that nobody re-made).
KillerYou have two thousand tests. How do you know the tenant filters are actually there, rather than the tests passing for other reasons?
They were mostly not proving it, and I measured that rather than assuming it. I deleted eighteen tenant filters one at a time and ran the suite after each: the two hand-written isolation files caught four, and one deletion turned nothing red at all. So the guard is derived from the code now — every scoped function called with a foreign scope, a syntax-tree pass requiring every query under app to name the company, and every mounted route driven signed in as one company. Asking per query rather than per function took it from seventy-eight of a hundred and seventy-six deletions to a hundred and one, which is not all of them.
Evidence. tests/test_tenancy_derived.py:1-42 (the eighteen-deletion audit and the function nothing caught, queries.py::passage_counts_by_version), :795-801 (78 of 176 to 101), :822 (AST_EXEMPT is empty), :840 (AST_QUERY_EXEMPT: 8 entries, 11 allowed queries, each with its reason). The seventy-five deletions the rule misses are mostly queries keyed on a row already resolved under the scope. No mutation harness was committed, so the 176-deletion campaign is a recorded measurement in that file's own comment rather than something a reviewer can re-run.
"What is the one thing you would rewrite" is in the five, and it is the same defect seen from the other end.
KillerTake the corpus to a hundred times its size. What gives first, and where exactly?
The claims read, and I can point at the line: the function that builds the source map loads every version the company owns, whole source text, on every change render. Today that is a hundred and two filings, fifteen megabytes; at a hundred times it is about one and a half gigabytes in memory to draw one page. Second is verification cost per citation, linear in the document rather than the corpus — nine milliseconds on fifty-three kilobytes, a hundred and seventy on a megabyte. Third is the retrieval fallback, a whole-corpus scan by design; the first two share one fix.
Evidence. app/state/claims.py:469-499 (_sources_by_version reads versions_for_company entire); app/verification/verifier.py:151-201; app/state/queries.py:258-277. Measured: data/real is 102 files, 15 MB; 8.9 ms on 53,132 chars, 168.7 ms on 1,024,536 chars. More filings do not make verification worse; bigger ones do. The first two need one scoped single-version read in queries.py, and the same edit turns the hash-drift check on.
"Who is this for" and "did you talk to real users" are both in the five. These four come after them.
CoreThe compliance officer owns the obligation register. Isn't she the buyer and the user?
She may well be the buyer. She is not the user: the register is a list of duties already agreed, and the expensive work is turning a new version of a proceeding into a judgement about whether one of those duties just moved. That is production, not consumption, and the analyst does it. The obligation owner is the one other role in the model and is not a second user — he approves what the analyst recommends, and that separation is a security control: the person who interprets a change should not be the one who signs off acting on it.
Evidence. docs/.ai/decisions.html:42 (ADR-01's second role, named as load-bearing and not a second user); app/state/routing.py:203. Role permissions read live from verbatim.db: analyst holds action.propose; obligation_owner holds action.approve and action.reject; admin holds neither. Self-approval is refused in code: can_approve reads authorship out of the audit chain, tests/test_policy.py holds it, and until ADR-91 no view called it. /actions calls it now — propose an action there and try to approve it and the answer is 403 naming the audit row. The limits: approving performs nothing beyond recording the decision; the default seed shows the happy path and no screen can arrange the refusal, so watching it needs VERBATIM_DEMO_REFUSAL=1 python -m app.seed (ADR-91 cost two); and where the demo lets you self-approve that is a disclosed downgrade rather than the design.
CoreWalk me through their day. Where exactly does this help?
Several open proceedings at once, in several states: months of quiet, then a draft order posts, then a short comment window, then a final order with a deadline attached. The moment I built for is narrow. A second version of something she has already read lands, and she does not want a summary, she knows the old one — she wants the delta in the source's own words, whether it binds yet, which of her obligations it touches, and who needs to see it this week. Today that is two documents side by side and a redline built by eye. This is a model, not an observation.
Evidence. docs/prd.html:60-63, and the section is headed "modeled, not yet observed" with its own honesty check at :61; docs/synthetic-user.html:61-62 — filings arrive as PDFs by email against a service list, often scanned, two copies of everything, one public and one confidential.
KillerYou wrote the persona, from filings you picked, for a product you built. Why is that worth anything?
Its agreement is worth nothing, and the page says so: it was assembled by the same mind, from a corpus that same mind chose, so it produces the pains the product already claims to solve. Two things make it non-zero. Every trait carries a file and a line in a real filing, so a reader can check the trait rather than trust me. And it earned its place by failing: asked to walk through last Tuesday it had nothing, so that question survived onto the real script.
Evidence. docs/synthetic-user.html:53 (bias on purpose — the corpus was selected for errata and version pairs), :66 ("A trait with no source does not appear"), :252-253 ("It will agree with the product too readily"); docs/prd.html:88-89; docs/user-research.html (the rehearsal cut four questions and produced the two marked "can hurt"). Traits that could not be sourced are tagged as guesses with what would kill them.
KillerOf the last ten things that landed on her desk, how many had an earlier version she had already read?
I don't know, and if the answer is two then my unit of work is aimed at the wrong moment. The product only makes a change when it has two versions to compare, and the first version of any proceeding has nothing to diff against — no branch for that in the design, the data model or the board. If most arrivals are cold, then cold start is not a gap in the product, it is the product, and the change-centric decision gets rewritten rather than patched. It is question two on my call script, marked as one of the two that can hurt.
Evidence. docs/prd.html:87-89 (cold start, and "It may be worse than a weakness"); docs/user-research.html:239-240 (question 2, tagged as able to hurt ADR-002). The question came out of rehearsing against the synthetic persona and is the sharpest thing that rehearsal produced.
CoreSo what would you cut?
The approval-route machinery — the part that decides what happens next, because nothing in the public record told me what happens next at any real company. Not routing: routing stays, because it is the part that refuses, walking change to obligation to owner and returning one of sixteen named refusal codes rather than a guess. On the seeded data almost every mapped change refuses to name a person anyway, because exactly two mappings are hard-coded as person-confirmed and a proposal cannot route. That is a quieter demo and it is the correct answer.
Evidence. app/state/routing.py:165-197 (the ROUTE_ codes), :203 — read live, len(ROUTING_REASON_CODES) is 18 and ROUTE_OK_CODES holds 2, leaving 16 refusals; :940-975 (only AUTHOR_ANALYST routes); scripts/seed_demo_gaps.py:71-74 (CONFIRMED_BY_A_PERSON is exactly two pairs); ADR-87 at decisions.html:904. Mapping totals belong to a fully seeded run — the working database holds one mapping row, so the total to quote is the one make seed prints. The cut costs the best-looking screen in the admin area.
HardHow do you know they want this and not just a better search box?
I don't know it — that is hypothesis two and nobody has answered it. What I can defend is the choice: search assumes the question, and she can only type where does this proceeding mention interconnection timelines if she already knows a change touched them. Her starting point is the opposite, so I built retrieval and refused search — full-text indexing is in the code, and there is no search box, no search screen and no search route in the sixty-three the app mounts. A ranked list puts a ranking in front of the evidence.
Evidence. docs/.ai/decisions.html:45-51 (ADR-02) and :689-696 (ADR-71: 366 ms scanning against 1.3 ms indexed on 8,707 passages); app/state/search.py:27-32; app.routes counted live — 63 routes, none matching "search"; docs/user-research.html:155 (H2). ADR-71's retrieval numbers are measured on the fully seeded corpus, not on the working database. The deterministic diff is a floor rather than a moat — a competent team rebuilds it in weeks, and the MRD says so. Filers do publish their own redlines; the Utah one invented a change and hid a real one.
HardName the one feature that does not map to a named user pain.
The approval-route engine — the largest body of code in the product, resting on the weakest evidence in the persona. The pain it claims to serve is chasing an approval by email, but the persona row underneath it, that the obligation owner is a distinct person who approves her work, is tagged assumed rather than grounded: internal approval process never gets filed, so a hundred and two public filings cannot show it. And the clock does not even tick unless someone sets an environment variable, which is off by default.
Evidence. Two and a half thousand lines of state machine plus fourteen hundred in the admin canvas — steps, timeouts, reminders, escalation, bypass: app/state/workflow.py is 2,471 lines, app/web/views/admin.py is 1,431. docs/synthetic-user.html:229-231 (tagged ASSUMED, and why: internal process is not filed). deploy/entrypoint.sh:116 and app/jobs/runner.py:211 — nothing runs unless VERBATIM_JOBS_ENABLED is "true". The citation verifier and the audit chain are not in the same position: both trace to grounded rows — a redline that lied, and a commission that replaced its own paraphrase with a pointer to the source.
HardWhat would you change?
The front door. Ingestion assumes somebody hands the system two versions of a proceeding, and my own call script carries a question I have never been able to ask — walk me through the last thing that landed on you, starting from how you found out it existed. If the honest answer is a colleague, or a law firm memo, or a phone call, then ingestion is not the front door at all. Second, how I spent the time: hour forty on an approval-route canvas, hour one on cold email. I would swap those.
Evidence. docs/user-research.html:236-237 (question 1 and what it settles: where the product's front door belongs); :81-86 (targeting was not the problem, the channel was). The best untried channel is a consumer-counsel reader — never the buyer, reads the same dockets under the same clock, likelier to answer inside a day — and one warm introduction is worth more than a forty-sixth cold message.
HardWhat did reading 102 filings teach you that you did not expect?
Three things, and the first two changed the design. The redline lies: a Utah errata redline shows a page reference of eighty-six where both the original and the clean copy read eight, a change that never happened, and it silently passed over one that did — so the wedge is not producing a redline, it is ranking one. Second, a diff cannot see what is not there: a Missouri corrected stipulation is fifteen lines longer than the original, the prose identical, and the added lines are the rate table. The document that sets the prices had been filed without the prices. Third, and this is why the whole product exists: in a Kentucky testimony the body says Brown BESS twice and an exhibit bound into the same PDF says Cane Run BESS.
Evidence. Verified by hand this session. ut-24-035-04-ellis-phase3-direct-testimony-errata-redline.txt:33 reads "86" where -original.txt:31 and -errata-clean.txt:31 both read "8". mo-ET-2025-0184-nonunanimous-stipulation-corrected.txt is 1,231 lines against 1,216; the diff shows the title, the date (7th to 9th), and Exhibit A — the Schedule LLCS pricing table at :1218-1232, absent from the original. ky-2025-00113-kollen-direct-testimony-original.txt:4600 and :4608 say "Brown BESS"; :13568 and :13570 say "Cane Run BESS". The product does not catch the missing rate table: the diff reports the changed title and date, and catching an absent attachment needs a comparison of the attachment manifest, which is not built.
HardYour strongest number is the Kentucky pair — a million characters, a hundred and twenty-seven apart, a hundred and forty-four changes. What obligations came out of it?
Close to none, and the reason is structural: that pair is intervenor testimony, and intervenor testimony does not place duties on a utility, it argues. My own gold set says it — the Kentucky corpus there is testimony, errata and cover letters, and it holds one duty across six passages against Georgia's ten across nineteen. So the diff is real arithmetic on a real filing and the thing downstream of it has almost nothing to bite on; the eight obligations in the demo are written against the invented Meridian docket.
Evidence. The pair is Lane Kollen's direct testimony in Kentucky, original against corrected. Re-measured this session: the two files are 1,024,409 and 1,024,536 chars, 127 apart; segmenting both and running app/diff/engine.py::diff gives exactly 144 changes — 134 modified, 6 added, 4 removed — in 0.38 s over 4,285 passages against 4,287. Scorer output: KY 6 passages, 1 obligation; GA 19 passages, 10 obligations. data/company_context.json holds 8 obligations, all Meridian. Kentucky was chosen because the filer published their own marked-up copy, which gave an answer key nobody here wrote — a verification decision, not a product one. The number proves the diff engine and nothing about the wedge; the better pair is a Georgia compliance-tariff order against its revision.
KillerWhy would they pay, and who signs the cheque?
I don't know. No buyer has been spoken to — my interview plan targets analysts, not the budget holder above them, and neither has happened — and there is no price in any document here. What I will commit to is the reasoning. The user is sold on time and on trust: does it read the docket correctly, and can she check a claim without redoing the work. The buyer is sold on risk: can the company show an auditor that a specific order was seen, assessed and acted on inside the deadline, with a record of who reviewed it.
Evidence. docs/mrd.html:619-625 (buyer versus user, and the open row saying which budget it comes from is unconfirmed); docs/synthetic-user.html:240-241 ("It cannot price anything"). A number invented at this desk would end up in the MRD and be indefensible. The MRD refuses a market-size figure for the same reason. Nothing in a public filing says what a seat costs.
All three are killers. Reproduce the scorer before the answer: .venv/bin/python -m app.evals.obligations --no-extractor.
KillerYour obligation gold set was labelled by a model. You are grading a model with a model.
Correct, and both files say so in the field: Claude Opus 5, single annotator, no adjudication. There is no second pass and no inter-annotator agreement number, and none can be computed from those files. Seventeen of the fifty-one passages carry an ambiguous flag, which is the labeller's own guess at contestability rather than a measured disagreement rate. So the honest claim is narrow: the set is internally consistent and its quotes still slice out of the source byte for byte. It is not a regulatory expert's judgement of what a filing obliges a utility to do.
Evidence. data/evals/goldset_ga_ky.json labeller field: "Claude (Opus 5), single annotator, no adjudication". data/evals/goldset_ut_mo.json labelled_by: the same phrase plus "regulatory text only". 25 + 26 = 51 passages, 9 + 8 = 17 ambiguous. Reproduced live. One live contradiction sits beside it: app/evals/obligations.py:10 still says the 51 passages are "each labelled by hand", and the data names a model. Fixing the set itself takes one regulatory-affairs professional relabelling a sample and an agreement number against the model's calls, which is not a build task.
KillerYour obligation scorer reports zero wrong assertions. That is your headline safety number. Defend it.
It is empty and I will not quote it alone: there is no extractor in this build, so the only run available was no-extractor, which records a refusal for all fifty-one passages. A system that asserts nothing cannot assert wrongly, so the blocking metric passed by asserting nothing, and correct assertions is zero on the same page. The scorer says so in its own output and tells you not to quote the exit code of this run. Twenty-three misses is the only figure in that run carrying information — it is the whole of what an extractor would have to earn back.
Evidence. Reproduced live: wrong 0, missed 23, correct 0, correct refusal 28, unadjudicated 0. No predictions file exists anywhere in the tree, and no code path reads a passage and says what duty it creates. app/evals/obligations.py — the flag's own banner reads "NO EXTRACTOR RAN" and "Do not quote the exit code of this run"; docs/.ai/eval-result.html. The run also prints "VERDICT: 3 of 4 metrics pass", and two of those three pass vacuously, which the scorecard labels. The metric can fire: a deliberately bad keyword baseline — shall, must, is required — was driven through it once and scored fourteen wrong assertions and exit 1. It was built in memory, never written to disk, and is not a product path.
KillerYou have two gold sets with different labelling rules and you score them as one pool. Show me that is safe.
I cannot, and it is the sharper problem of the two. The Georgia-Kentucky file has a rule the other does not: a passage that restates a duty created somewhere else is labelled false unless this document is the instrument imposing it. The Utah-Missouri not-an-obligation list has six entries and no restatement clause at all, so the same sentence can score true in one file and false in the other. And the loader never reads either rule field, so nothing in the scorer can notice: the flattening function pools both into one tuple, and the counts are arithmetic over two definitions.
Evidence. data/evals/goldset_ga_ky.json labelling_rule (the restatement clause); data/evals/goldset_ut_mo.json labelling_rules.not_an_obligation — six entries, and the string "restat" appears nowhere in that object. app/evals/obligations.py:321-330 (GoldSet carries name, path, jurisdictions, known_limits, passages — no rule field), :442-455 (all_passages flattens every set into one tuple). The definitions are not close enough to pool: the verifier tests equality rather than similarity everywhere else, and pooled labels defended on similarity contradict ADR-03. The fix has two steps — make GoldSet carry its rule text and refuse a pool whose rules differ, then relabel one set against the other's rule. The first is an afternoon; the second needs an expert.
CoreYou have no CSRF token anywhere in this application. Every admin form is forgeable.
There is no token, correct — SameSite=Lax on the session cookie is the whole defence, and I wrote that in the modules that own forms rather than leaving it to be found. Lax stops a cross-site POST from carrying the cookie, and does nothing against an attacker who can get script onto my own origin. The login form has no token either, so somebody can make a victim's browser sign in as the attacker. There is a per-account lockout — five wrong passwords, fifteen minutes — but nothing throttles guesses spread across many addresses.
Evidence. app/web/deps.py:78-79 — the comment says Lax "is the whole of the CSRF defence on this build — there is no token"; app/auth/sessions.py:45; app/auth/sessions.py:101-102 (MAX_FAILED_ATTEMPTS = 5, LOCKOUT 15 minutes). The same concession is written in four view modules: auth.py:28, users_admin.py:77, shares_admin.py:67, permissions.py:74. Per-source throttling belongs at the edge and this build has no edge. Ranked by what a regulated buyer blocks on rather than by what is easiest: SSO first, then per-source throttling, then the CSRF token, then encryption at rest.
CoreYour demo signs anybody in as an administrator with one click and no password. Walk me through what that account can do.
It holds four administrative codes — manage users, invite, manage the workflow, set the threshold — and the one-click login is my call, ADR-84. One unauthenticated POST to the demo login returns a session, and it is not a bypass: the handler calls the same login function with the seeded password, supplied by the server instead of by somebody's fingers, so there is no second door. The full cost is the handoff invite path, where a visitor can invite an address they control, accept it, and hold approve, with nothing rate-limiting that. What is not at stake is confidentiality.
Evidence. app/web/views/auth.py:305-350 (demo_login_post, and the docstring on why it is not a bypass). Role permissions read live: admin holds eight codes, of which user.manage, user.invite, workflow.manage and threshold.set are the administrative ones — the other four are reads every role has, and admin holds neither action.approve nor action.reject. app/state/invites.py:35-49 and :2222-2233 (CEILING_WAIVED_BY_KIND names the handoff hole in writing); ADR-84 at decisions.html:853-860, including its own correction that "no way back" was false when written; make reset-demo puts the tenant back. Provisioning is ceiling-bound and cannot escalate; the handoff path deliberately can, and it writes an audit row saying so every time. Turning the demo off is not a cure: security.html:147 says in its own words that the environment variable hides the panel and does not remove the accounts. The panel does not ship to a real customer.
HardYour own stylesheet says a text colour fails accessibility. Explain why it is still in the build.
Because darkening that token touches every secondary label in the product and I ran out of hours. The third ink tone on the masthead does not clear AA: sampled every pixel inside the nav-link box on a real page at 1440 by 900, best pixel four point four nine to one, worst three point eight five, where AA wants four point five. The figures I had published — 4.57, and before that 16.15 — were computed against the flat fallback token, or against an ink colour that appears nowhere in the stylesheet. The pane is glass, so a rendered figure there is a range, not a number.
Evidence. app/web/static/verbatim.css:161-183 — the sampled figures at :169-170, the correction at :171-175, and 16.73 / 8.26 / 4.57 labelled as token arithmetic against --glass-solid at :176-182. --ink-3 is #67747a at :87. The first three measurements were of the token, not the pane: the pane takes the colour of whatever is under it, and something always is, because content scrolls beneath a sticky masthead.
KillerSeveral times in this repository you shipped code that nothing called. Why should I believe there is not another one in there right now?
There is. One is still open and one closed today. start_run is the open one; can_approve was the other until today, and how it got fixed is the more useful half. start_run, the function that begins an approval run, still has no caller in the application, so no run ever starts. can_approve had none for a day — an independent review found it before I did — and two attempts to wire it into the escalation queue were reverted, because that screen belongs to the analyst and gate 2 demands a permission the analyst does not hold. The real gap was upstream: there was no screen where an action could be proposed at all, so the gate had no decision under it. ADR-91 built the screen; app/web/views/actions.py::decide is now the caller. What changed is how I look for these: the wiring test used to assert five hand-written paths while two view modules with nine routes between them were mounted nowhere, and rewritten to read the route table it caught three unmounted routers on its first run.
Evidence. grep -rn can_approve app/: hits are app/auth/policy.py docstrings, its definition at :531, a comment at app/web/views/admin.py:1219, and the call in app/web/views/actions.py. start_run is defined at app/state/workflow.py:1744 and called only from tests. tests/test_app_wiring.py derives the check from app.routes. docs/security.html:131 records the gap and where it was closed, in its own words. can_approve reads authorship out of the audit chain. Every one of these was green over more than two thousand tests, because a test proves capability and says nothing about whether a thing is wired or seeded. The general rule the fixes share: derive the guard from the artefact instead of listing it — the wiring test reads the route table, the tenancy sweep reads the syntax tree — because a list a person maintains cannot catch the module that person forgot, and "every read" in a docstring is an aspiration until something counts the readers.
HardNinety decision records in three days. How much of this did you write?
Most of the lines are the model's, and the submission record says so before anybody asks — fifty-seven thousand lines under app, fifty-two thousand under tests, seventy-six commits in three days. What is mine is the architecture, the plan, and every position an ADR takes. The rows worth reading are the rejections: a similarity threshold in the verifier, raw similarity as diff confidence, a fixed-width window in the occurrence check, an analytics screen, a percentage computed over five items. One runs the other way — I asked for a ninety-nine percent coverage target, the model refused it, I overruled the refusal, then cut the target myself.
Evidence. git rev-list --count HEAD gives 76 at a5c20cb. wc -l: 57,435 lines under app/, 52,435 under tests/. docs/.ai/decisions.html carries 90 unique anchors, adr-001 to adr-090. docs/submission.html holds the "what the AI wrote, and what I rewrote or rejected" table. The three decisions that are mine to defend: citation verification on the read path with no stored verdict; the model judges materiality and never finds changes; a refusal carries a named code rather than an empty result. Each has its rejected alternatives written down the day it was made.
Every limit below is a fact about the system, stated here rather than left to be found. Each one is checkable in the tree.
Nobody has been interviewed. Zero. Forty-five messages, four bounces, no replies. User empathy is the weak dimension in this submission and no amount of further engineering moves it.
The persona is invented and labelled so. Its agreement is worth nothing. Its traceability is worth something. Never offer it as a substitute for a conversation.
The gate checks the words, not the reading. Verified means the words exist where the claim says they do. A fluent, correctly-cited, wrong reading passes.
One control is built, tested, and not wired. start_run begins an approval run and nothing in app/ calls it. can_approve was the second until ADR-91 gave it a screen — say that it took a whole missing decision rather than a line, and that a review found it first.
Drift is caught on cited spans only. The hash check exists, is tested, and is not on the read path. Edit twenty characters elsewhere in the same filing and the claim still asserts.
There is no CSRF token. SameSite=Lax is the whole defence. The login form has none either.
The chain is tamper-evident, not tamper-proof. Anyone who can rewrite the whole database recomputes every hash forward. Closing that needs the head published outside, and it is not built.
The evals measure the deterministic half. There is no eval on the model path, and the obligation scorer's zero comes from a build with no extractor.
The live model path is barely exercised. One deliberate call, one accidental one that cannot be reported, and no counter anywhere for the assistant's calls in production.
The approval engine rests on the weakest row in the persona. Largest body of code, tagged ASSUMED, and its clock is off by default.
The diff does not handle restructures. It refuses to present one as settled. That is the claim, and it is narrower.
Four stale sentences, ranked by what a reviewer hits first. They are in the repository itself, not on this page.
1. docs/submission.html is stale on its own front page. It says 43 commits at :40 and :81, fifteen routers, 50 routes and 18 templates at :74, and "The repository is PRIVATE" at :82. The tree has 76 commits, 17 routers, 63 routes and 24 templates, and the repository is public. This is the first document a reviewer opens.
2. app/evals/obligations.py:10 says the 51 gold-set passages are "each labelled by hand". Both gold-set files name a model. The data is honest; the docstring is not.
3. docs/security.html:147 says "a password the login page publishes". Commit 9ac91a7 took the password off the page. The rest of that line — the environment variable hides the demo panel and does not remove the accounts — is still true and still load-bearing.
4. app/state/mapping.py:4 says routing "refuses in fifteen named ways". ROUTING_REASON_CODES holds 18 codes of which 2 are OK codes, so the count is sixteen. docs/.ai/eval-result.html is stale in the same way: it says goldset_ut_mo does not name its labeller, and it does.
One more, outside the four: docs/.ai/state.json was generated when the repository had 39 commits, and its own header claims it is authoritative wherever prose disagrees with it.