What was built, what was reused, what the AI wrote versus what was rewritten or rejected, and what broke. Every row names a file, a commit, a route or a test; where a claim could not be checked against the repository, it says so.
Verbatim takes two versions of a regulatory proceeding and tells a regulatory affairs analyst what changed, which of their company's obligations it touches, and who has to act — and it refuses to say any of it without showing the words in the source. Every claim carries a citation that is re-read from the stored bytes on the request that renders it, so a claim whose source has moved withdraws itself rather than repeating what used to be true. Around that sit the parts an enterprise asks about before it asks about features: named roles and per-permission grants, an approval route an administrator shapes to their own organisation, segregation of duties, an append-only hash-chained audit log where a reversal is a new row rather than an edit, tenant isolation through a single chokepoint with tests derived from the code rather than a maintained list, security headers on every response, and an accessibility sweep against WCAG 2.2 with contrast measured from rendered pixels rather than from tokens.
The running product is at verbatim.citelocal.ai, and the home page carries a video walk-through of the product in use — the fastest way to see what this record describes. The source is at github.com/jsahasi/verbatim.
| Field | Value |
|---|---|
| PRD | docs/prd.html. Living document. The user-research section is the
weak one and says so: no interview has happened, so the persona in ADR-001 is still a hypothesis.
docs/synthetic-user.html exists and is labelled at the top as not evidence and not a
transcript — it is a question generator, and no line of it may be moved into the PRD as a
finding. |
| TDD | docs/tdd.html, whose Part A carries the module
boundaries, with docs/security.html for the threat model. Decisions and their rejected
alternatives are in docs/.ai/decisions.html — 95 entries at the time of writing, ADR-001 upward with no gaps, counted with
grep -c 'data-adr=' docs/.ai/decisions.html rather than remembered; this row said 63 for
several days after it stopped being true, which is why the command is printed beside the number.
docs/scalability.html carries the measured cost of the diff path and the design that
follows from it, and docs/structure-measurement.html carries a negative result:
a document-tree parser, measured on the real corpus and withdrawn. Five of the early ones had gone false
and were corrected in place with their original text kept beneath (295c40d). |
| Prototype | Local: make run, then http://localhost:8000.
Nineteen routers, 70 routes and 26 page templates are served — project list, proceedings, one
proceeding, one change, the review centre, the escalation queue, the approval route, the actions
screen, Clarke the assistant, the "how it works" page, share links, invitations, users, Integrations
and the feedback queue. Read off the assembled application
(from app.main import app; len(app.routes)), not counted by hand. Hosted:
verbatim.citelocal.ai serves the product behind the marketing site
(b5c894b). Checked from here over the internet: /healthz answers
corpus_loaded: true with 2 proceedings, 5 versions, 171 changes, and
/projects answers 303 to /login rather than 404. |
| Source code | https://github.com/jsahasi/verbatim. 183 commits at the time of
writing (git rev-list --count HEAD), not squashed. The repository is PUBLIC. Confirmed here, not assumed:
gh repo view jsahasi/verbatim --json visibility returns PUBLIC, and an
unauthenticated request for a file blob answers 200. It was private until the history had been
checked for secrets: making a repository public exposes every commit ever made, not the working
tree, so a key committed once and deleted later is still readable. .env was absent
from all of it. This is the one item on the
checklist that stops a reviewer at the front door. |
| Access notes / credentials | None needed for the deterministic path — ingestion,
normalization, citation verification, occurrence identity, diff, draft-versus-final, tenant scoping
— which runs with no API key and no network, and so does make eval. A key buys three
things and none of them sits on the verification path: Clarke the assistant
(app/chat/), the claim proposer (app/interpretation/propose.py), and the
assistant on the "how it works" page (app/explain/). All three
decline and say so without one. Secrets live in one gitignored .env (ADR-014), loaded
by app/config.py before anything reads the environment;
.env.example lists all twenty names — sixteen the application reads, plus four Gmail
values at the foot marked as belonging to the outreach script and not to the product.
A second key exists now and it is optional. OPENAI_API_KEY buys
better retrieval on the "how it works" page and no extra capability: without it the same document
sections are ranked by bm25 and every reply says so in a sentence of its own. It has been
used: on 2026-08-11 the whole corpus was embedded against text-embedding-3-small —
145 chunks, two requests, 51,854 tokens by the provider's own count, about a tenth of a cent — which
is the first live embedding call this code has ever made. The provider moved from Voyage for that
reason alone: a key for one is in the team's hands and a key for the other never was, so every
constant in the file was a documentation reading checked by nothing. ADR-104 carries it. Every test
still drives an injected fake and the suite makes no network call. |
| What I personally built | The decisions and the plan; the AI wrote most of the lines. See the two sections below. |
| What I reused | docs/best-practices.html — 27 principles carried from a
peer project, and 31 at HEAD: §28 to §31 were born here, out of this build's own
failures — a verified fact with a shelf life, built-is-not-connected, prose goes stale and nothing
catches it, and a guard's scope is itself unguarded. Plus the append-only-audit and
escalate-rather-than-guess patterns from that same project, and ten runtime packages. No code was
copied from it. |
| What the AI wrote, and what I rewrote or rejected | Table below. The rejections are the rows worth reading: a similarity threshold in the verifier, raw similarity as diff confidence, a fixed-width window scan in the occurrence check, an analytics screen, a percentage computed over five items, a fifty-value secrets file, and a role vocabulary built on job seniority. One row runs the other way and is the more useful one: I asked for a 99% coverage target and the AI refused it. I overruled the refusal, then cut the target myself. |
| What broke and how I debugged it | Table below, 22 rows. Thirteen are logged with their
guards in docs/.ai/findings.html; the other nine appear here first, including the three
that cost the most. Five landed on the last day and are the sharpest: nothing loaded .env, the
deploy ran no migration, init_db() dropped every table against its own docstring, a
share link redirected its own bearer token into a query string, and the test written to catch an
unmounted router could not catch the next one. |
I chose the architecture and wrote the plan; an AI coding tool wrote most of the lines under it, in TDD waves against tests specified before the implementation. The claim is not that I typed it: every load-bearing decision has an ADR written before the code, with its rejected alternatives, and I can name where the tool was wrong.
| Module | What it does | Guarded by |
|---|---|---|
app/ingestion/ingest.py | Segments a version into passages with stable character offsets; retains a SHA-256 of the source. | tests/test_ingestion.py (4) |
app/text/normalize.py | The single normalization used on both sides of
every citation comparison. Folds quotes, dashes, ligatures, whitespace; deletes soft hyphens;
rewrites hyphenated line breaks across the seven characters Unicode counts as a line boundary, form
feed included, which is the one a paginated filing is guaranteed to carry. Never folds digits,
units, dates or case. A character is protected from NFKC when its folded output begins with a
decimal digit and the character is not itself a digit — a rule tested against the folding
rather than read off a tag list, because the tag list it replaced was wrong by 210 characters
(34eff78). normalized_projection() carries a map from each normalized
character back to the raw characters that made it. | tests/test_normalize.py (30) |
app/verification/verifier.py | ADR-003, the depth area. Re-reads the
stored source at the cited offsets and compares for equality after normalization, not
similarity. A repeated quote must state which occurrence it relies on.
verify_citation_for_version() loads the version the citation names, through the tenant
chokepoint, so the citation cannot be verified against the wrong text — and refuses outright when
the stored bytes no longer hash to DocumentVersion.source_sha256, with its own reason,
so a tampered version is never confused with a quote that does not match
(ec1c4bf). | tests/test_verification.py (23),
tests/test_occurrence.py (9) |
app/diff/engine.py | ADR-004. Deterministic alignment and typed changes. Alignment confidence is capped when the section label disagrees, so a renumbering escalates. | tests/test_diff.py (11) |
app/interpretation/action.py | ADR-005. Draft and final have separate
action vocabularies and separate code paths; Change.status is copied at write time, not
joined at read time. | tests/test_draft_final.py (5) |
app/state/queries.py, app/state/claims.py | Every read that returns content is tenant-scoped through one auditable place. Claims are verified at read time and withheld rather than shown when they fail. | tests/test_isolation.py (3),
tests/test_passage_isolation.py (5), tests/test_claims.py (19) |
app/state/audit.py | Append-only audit with two independent mechanisms: a
before_flush guard against application code, and a SHA-256 chain against out-of-band
rewrites. Events now carry actor attribution. Each mechanism is honest about what it does not
do. | tests/test_audit.py (8), tests/test_audit_v2.py (25) |
app/state/identity.py, app/auth/ | Users, roles and
permissions, with scrypt password hashing and hashed session tokens; the session guard is installed
once in app/main.py rather than decorated onto each route, so a screen added next month
is protected by existing. The login path this was built ahead of now exists, and a person signs in
over the internet. | tests/test_identity.py (33),
tests/test_login.py (33), tests/test_policy.py (28),
tests/test_provisioning.py (51) |
app/config.py | Loads .env into the process once, before
anything reads os.environ. Fifteen lines of standard library rather than a dependency.
The real environment always beats the file, and it never logs a value. It exists because nothing did
this, so every model path announced itself as off while a valid key sat on disk — see the break log.
| Nothing. No test file imports app.config or calls
load_env. The guard that would matter is the one asserting the environment beats
the file, because reading it the other way round means a container cannot override its own image. |
app/state/migrate.py | Asks SQLAlchemy what the models declare, asks the
database what it has, adds the difference. Never drops, renames, retypes or backfills; refuses a
NOT NULL column with no default and exits non-zero, so a deploy stops rather than starting against a
schema it cannot read. deploy/entrypoint.sh runs it on every start. | Nothing.
No test imports app.state.migrate; docs/.ai/resume.md records it at 0%
coverage. It is the module that stands between a deploy and an unreadable live schema, and
tests cannot easily reach it: a test builds its schema from today's models, so it never meets
yesterday's database. The fixture it needs is an old schema built on purpose. |
app/chat/ — agent.py, tools.py,
engine.py, persona.py, pills.py | Clarke, the
assistant. The only place in the product where a model reads the record, and it reads it through
tools that go to the same tenant-scoped queries the screens use. No key, no model: it says so and
answers nothing. engine.py is a name rather than a layer — it exists because the view
resolved app.chat.engine.answer and the engine shipped as
app.chat.agent.run_turn. | tests/test_chat_surface.py (65),
tests/test_chat_tools.py (49), tests/test_chat_agent.py (43),
tests/test_chat_schema.py (19) |
app/pipeline.py, app/seed.py,
scripts/ingest_real.py | Ingest and diff end to end; load the corpus and the
workspace from the two data files, inventing nothing. ingest_real.py is how a real
filing gets in: it reads a provenance record from data/real/, keeps the source URL on
the version so every claim links back to the commission's own copy, and loads the Kentucky
2025-00113 pair — 1,024,409 characters against 1,024,536, 144 changes in 0.78
seconds. | tests/test_pipeline.py (17),
tests/test_seed.py (20), tests/test_seed_workspace.py (23),
tests/test_source_links.py (44) |
app/state/projects.py, app/state/review.py | The workspace AI Fund demoed: projects, threads, work plans, scheduled runs, superseded-never-edited knowledge, and a review centre whose synthesis refuses to compose a take that does not record its withheld count. | tests/test_projects.py (26), tests/test_review.py (19) |
app/web/views/, app/web/templates/ | Nineteen routers, 70
routes, 26 page templates (git ls-files 'app/web/views/*.py' and
from app.main import app; len(app.routes)). The citation chip opens the stored source beside the claim,
scrolled to the cited offsets with the span marked. The project list and the review centre are
mounted and reachable since 94ef20f; the path collision that kept them off is
resolved. | tests/test_change_view.py (18),
tests/test_screens.py (24), tests/test_app_wiring.py (10),
tests/test_tour.py (23). Still no test file of its own for
projects.py (872 lines) or review_centre.py (1,149 lines). They are
now exercised through test_app_wiring.py, which signs in and requests every screen, and
resume.md puts review_centre.py at 74% — the lowest module in the
repository. |
The workspace built after the engine — app/state/: sharing.py,
invites.py, workflow.py, sources.py, rollback.py,
routing.py, retention.py, feedback.py; plus
app/notify/ and app/interpretation/propose.py | Share links, invitations, the approval route, where filings come from, reverting an event, routing to the person who owns the obligation, retention windows, the feedback queue, mail, and the claim proposer — the only other module that calls a model. | tests/test_invites.py (63),
tests/test_feedback.py (60), tests/test_sources.py (54),
tests/test_workflow.py (53), tests/test_workflow_editor.py (44),
tests/test_source_links.py (44), tests/test_sharing.py (41),
tests/test_share_registry.py (40), tests/test_rollback.py (38),
tests/test_propose.py (37), tests/test_routing.py (35),
tests/test_retention.py (32), tests/test_notify.py (32),
tests/test_users_admin.py (52), tests/test_schema.py (18) |
app/evals/ | make eval. A scorecard over the deterministic
spine, scored against data/manifest.json. Five metrics, exit 1 on any blocking failure,
so it is a gate rather than a report. It prints raw counts and withholds a rate below a sample of
ten, and its caveat names the sample size and what it cannot support. Run in this pass: 5 of 5
pass, exit 0. | tests/test_evals.py (37), added at ac6a6fd. The rule
that matters most — refuse a rate below ten independent samples — is pinned, because a harness that
quietly reports a percentage off six items is worse than no harness. make test still
does not run make eval. |
data/ | Two corpora, each labelled for what it is (ADR-40).
Synthetic: a three-version proceeding with the traps built in — a material change,
a cosmetic one, a final-only addition, a moved deadline, a section restructure, and one sentence
repeated nine times. manifest.json is ground truth, computed independently and re-read
against the bytes. Real: data/real/ holds 102 public filings from
eight commissions, each with a provenance record naming its source URL, docket, filer, filing date,
retrieval time and SHA-256 (ls data/real | wc -l gives 204 — a text file and a
provenance file each). | Every test above asserts against the manifest rather than against whatever the code produces. The real corpus is not the eval corpus and must not become it: its answer key is the filer's own marked-up copy, which exists for three families and not on demand. |
app/evals/model.py, .github/workflows/ | The model
path scored, and the first CI this repository has ever had. The scorecard keeps fabrication
— an assertion whose citation the harness cannot re-read — apart from a miss, because the first must
be zero and the second is survivable, and one averaged number would report a working gate as a
failing one. CI runs make test and make eval, the two commands the README
promises, rather than a hand-rolled pipeline that could stay green while those broke. | tests/test_model_evals.py.
It also fails the build if --send ever appears in a workflow, if either make target
leaves the Makefile, or if any ADR number cited anywhere in the code, the Makefile or the README does
not resolve in decisions.html — that last guard exists because this harness's own header
once cited a decision number one past the end of the log. |
scripts/bench_diff.py, docs/scalability.html | The diff path
measured on a named machine, and the design that follows. Linear to 52.6 million characters;
quadratic on a run of repeated lines, which is an ordinary tariff sheet. The two findings worth the
page: the constraint is normalization at 87% of the call, not the matcher at 1.5%, so anybody tuning
SequenceMatcher spends a day and finds nothing; and partitioning — the obvious fix — ran
1.8 times slower and does not touch the quadratic case at all. It also turned up a live
defect that is documented and unfixed. | tests/test_diff_scale.py. One guard is
unusual and worth reading: it fails if a published number falls below what the pipeline measures, and
another watches where an allocation happens rather than how large it is, because two figures
on that page had already been wrong through a region being narrower than its label. |
app/diff/structure.py, docs/structure-measurement.html | A negative result, kept as the deliverable. A document-tree parser, written to fix a proxy that treats any leading integer as a section number and fires on two passages in five. Wired in, measured on the real corpus, withdrawn: 27 of 102 filings parse, and the branches it was built for fired zero times while 112 confidences moved on a parse failure withholding the cap. A refusal raising a score is absence licensing certainty. | tests/test_structure.py, including a test that
fails if anything under app/ imports the module again without that page
changing. |
app/state/alignment.py, app/chat/budget.py | Two smaller ones with the same shape: a question the product asked and did not keep, and a rule written before the feature it governs. A reviewer can now confirm, dispute, or say "cannot tell" about a doubted pairing — hash-chained, never overwriting the computed number, and "cannot tell" leaves the caution standing. The context budget is uncalled on purpose: multi-turn does not exist, and the policy every chat product reaches for evicts the offsets that make a claim checkable. | tests/test_alignment_decision.py,
tests/test_context_budget.py, and
tests/test_history_needs_a_budget.py, which walks the
syntax tree so the budget cannot be bypassed by a one-line edit. |
app/explain/ | The "how it works" page answers from this project's own PRD, MRD and TDD (ADR-095) — chunked by heading, embedded where a key is set and ranked by bm25 where it is not, with the fallback announced in the reply rather than in a log. Every citation in an answer is re-read against the file on disk before the answer is shown, and one that fails is marked unverified rather than dropped. Conceded rather than counted as product: this is built for the person reading the submission, not for the analyst in ADR-001. | tests/test_explain_documents.py,
tests/test_explain_tabs.py,
tests/test_explain_ceiling.py. No test has ever made a
live call to the embedding provider, and neither has anything else. |
The reviewer's command, run as a reviewer runs it. make fresh-check clones the repository into a temporary directory and runs the suite there, so nothing uncommitted can make it pass: 2,785 passed, 1 xfailed in 307 seconds, exit zero, "fresh clone: tests pass". Read the two halves of that sentence apart. The count is from a straight run of make test in this working copy on 2026-08-11 — 2,785 passed, 1 xfailed in 350.73 seconds — and it agrees with docs/.ai/state.json, which is generated by running the suite. The fresh-clone run behind the 307 seconds was 2026-08-08 at commit ba5965b, when the suite was 303 tests smaller, and it has not been repeated since the five branches landed. Substituting today's count into an older run's sentence is exactly the kind of quiet re-dating this document exists to refuse, so both dates stand. It has not always: an earlier entry in the break log records a fresh clone failing where the working copy was green, because a feature had landed half-committed. That is why the check exists.
The per-file counts above were read off
pytest --collect-only -q, not carried forward. Coverage is 92% of 10,359 statements
(804 missed), recorded in docs/.ai/resume.md at commit 2b10bb6; the lowest
modules are review_centre.py at 74% and migrate.py at 0%. The coverage tool
is installed in the virtualenv and is not in requirements.txt, so a reviewer
who clones this repository can run the tests and cannot reproduce that number.
| Reused | From where | What was not taken, and why |
|---|---|---|
docs/best-practices.html — 27 engineering principles, each with the failure
that taught it | A peer project of the author's | Nothing was dropped, and §28 was added here from this build's own mistake — an outreach address verified against a 2021 filing and wrong by 2026. At HEAD the file carries 27 numbered principles and §28 appears in the changelog rather than as a numbered section of its own. Sections 26 and 27 are load-bearing for Verbatim: a fallback must announce itself, and a derived corpus migrates all at once. Three separate defects here were fallbacks that announced themselves and announced something untrue. |
| Two design patterns: the append-only audit with a hash chain, and escalate-rather-than-guess (ADR-006) | Same project | Patterns, not code. ADR-006 exists because that project's embedding provider degraded silently and ranking ran on noise for months while every check passed. The equivalent here would be an impact mapping the model was unsure about, presented as settled. |
| Five secrets: one model key, four Gmail OAuth values | Same project's
.env | The other forty-five. That file carries production brokerage keys, a health-data encryption key, a database password and a source-control token. Copying it wholesale was one command and would have widened this repository's blast radius by every one of them, for no gain (ADR-014). |
| Ten pinned packages — eight runtime, led by FastAPI, uvicorn, SQLAlchemy, pydantic and Jinja2, plus pytest and pytest-asyncio | PyPI, pinned to minor versions | No vector
database (SQLite ships FTS5 and retrieval is lexical first, ADR-008), no migrations tool, no
frontend toolchain (ADR-012), no password library (hashlib.scrypt is in the standard
library), and no python-dotenv — app/config.py is fifteen lines of
standard library against a file whose format we control. requirements.txt lists each
omission with its reason. Two corrections since this was written. The
schema is no longer one version: app/state/migrate.py is a hand-rolled additive
migration, still not Alembic, and the file says why. And anthropic is no longer a
dependency nothing imports — app/interpretation/propose.py and
app/chat/agent.py both import it, inside a function, so nothing pays for the SDK at
import time. The pin moved from 0.39 to 0.120 to match what the callers send.
requirements.txt still names propose.py as the only importer, which is one
importer short. |
Nothing in data/ | — | The synthetic corpus is authored: every
name, docket number and commission is invented and flagged as invented in the manifest. Since
f302835 the repository also carries 102 real public filings under
data/real/ — retrieved, hashed and traced to the commission's own URL, never rewritten.
They are public record, not any customer's data, and each haul was re-checked by a skeptic that
re-fetched the source and recomputed the hash. Clean, well-paragraphed prose was treated as the
signature of something generated rather than extracted, so it was the finding to chase and not the
reassurance. |
Using an AI coding tool is expected; letting it choose the architecture is not. Each row names who decided the approach. The rows where the tool was overruled are the ones that carry information.
| What | Who decided the approach | What the AI produced | What I changed or rejected, and why |
|---|---|---|---|
| Citation verification (ADR-003) | Me, in decisions.html, before any code existed (commit 634d559) |
app/verification/verifier.py as specified: re-read the source at the offsets,
compare after normalization. |
Rejected the obvious implementation. The reflex answer — and the one a model reaches for — is a similarity threshold. A threshold accepts paraphrase, and a paraphrase is exactly what an auditor will not accept, so it defeats the point of having a gate. The verifier compares for equality. The module docstring says so, so a later editor cannot soften it by accident. |
| Occurrence disambiguation | Me, written into the plan as task 5 before implementation (b70d688) |
A verifier that passed a citation quoting the right words at real offsets while describing a different occurrence of them. | Caught by design, not by luck. The corpus was built with one sentence repeated three times per version, nine in all, precisely so this hole is testable rather than theoretical. Text equality is necessary and not sufficient: a repeated quote must now state which occurrence it relies on or it does not verify. |
| Finding the occurrences at all | Me, after a build agent reported the hole in its own work | A raw substring search with a fixed-width window scan behind it: normalize a slice of raw text at each offset and compare. | Rejected as wrong, not as slow. The scan assumes normalization
preserves length. Collapsing whitespace does not, deleting a soft hyphen does not, expanding a
ligature does not — so on PDF-shaped text it finds nothing, occurrence_count() answers
zero, and the repeated-boilerplate guard switches itself off in silence. That is a fallback that
does not announce itself, best-practices §26 exactly. Replaced with one normalized projection
carrying an index map back to raw offsets.
The replacement was wrong too. The new docstring claimed "O(n), not O(n·m)". The projection underneath is genuinely linear — flat at 171ns per character from 12.5K to 400K — but _spans_of was not, because the fix had
added a re-normalization per hit, and hits scale with n. Measured: 1.29s on a 40,000-character
document against a 200-character repeated quote, and 235ms on a 144KB filing full of rule lines
against a ten-underscore quote that hit 102,000 times. The re-check was load-bearing — it is what
stops a hit beginning part-way through an expanded character — so it was replaced rather than
removed: a hit is valid exactly when it shares no raw offset with the character either side of it,
and the projection already carries both arrays. Two integer comparisons instead of a
re-normalization (ec1c4bf). A performance claim in a docstring is a claim like any
other, and this one was written by the same fix that broke it. |
| Normalization strength | Me: fold what a tool could have changed, never what could carry meaning | NFKC applied to the whole string, against the module's own docstring. | Narrowed, rather than amending the docstring to match the code.
normalize("20²") returned "202" and "½" became three
characters. A footnote marker turning into a digit inside the function whose job is to preserve
value is not a rounding error. The other choice on the table was to weaken the docstring, and that
would have been the product's own failure mode: change the claim until the evidence fits.
The first fix listed three decomposition tags and the docstring called them "the three tags where folding rewrites a number". That sentence was false by 210 characters. normalize("20①") returned "201", and a circled digit is
a footnote marker in exactly the way a superscript is — so the class stayed open under prose saying
it was shut. The obvious repair was to add more tags to the list. That was rejected: a tag list
cannot express "would this fold rewrite a number", and protecting <font> while
folding <wide> would give the module two answers to one question, since
FULLWIDTH DIGIT TWO and MATHEMATICAL BOLD DIGIT TWO are both the digit two in another face. The axis
was wrong. A character is protected when its NFKC folding begins with a decimal digit,
unless it is a digit itself — testing the output rather than the label. 150 characters newly
protected, none unprotected, and a future Unicode version adding markers is covered without editing
a list. Proof of no collateral damage: the old module and the new one swept over all 0x110000 code
points in five contexts; 755 outputs differ and every one is explained (34eff78). |
| Diff alignment confidence (ADR-004) | Me: low confidence must escalate | Confidence reported as raw text similarity. | Rejected and inverted. The corpus restructure — Section 6 becoming subsection 5.4 — scored 0.944 and presented itself as a confident match. That is backwards in the dangerous direction: a renumbering is dangerous because the words barely move, so similarity runs high exactly when structural identity has changed. Confidence is now capped at 0.5 when the leading section label disagrees. Three regression tests, one of them proving an ordinary in-section edit keeps its true score rather than escalating everything. |
| Test standard | Me: 99% coverage on every module | An argument against the number, and a substitute standard (ADR-38). | I asked for 99%. The AI refused it, I overruled the refusal, then I cut the
target myself. The refusal was right about
line coverage: it measures execution, not verification, and the line that rejects a fabricated
quote can be green while nothing asserts that it rejects anything. Its substitute was branch
coverage on the four deterministic load-bearing modules, mutation testing, property-based tests for
the invariants, and evals rather than coverage for the model path. What the refusal got wrong is
that it was declining to measure anything at all: none of that substitute had been built, so
"coverage is the wrong number" was doing the work of "we do not know". I overruled it and asked for
both. A coverage tool went in, the answer became 92% rather than an opinion, and once the remaining
target was priced — three to six hours of tests written to move a figure, against a submission
whose thinnest dimension is user research — I cut it. Mutation testing is still not built. The one
module I would cover on merit is app/state/migrate.py, which stands between a deploy
and an unreadable live schema and has no tests. |
| The eval harness | Me: counts, not adjectives, and no model in the loop | A scorecard over the deterministic spine — app/evals/, five metrics, an exit code
so make eval gates rather than reports. |
Accepted, with one rule forced into the reporting layer. The corpus
labels five changes. Any percentage over five items is false precision, which is the exact sin this
repository keeps catching, so app/evals/report.py owns a rule no future metric can
route around: print raw counts with their denominators, print a rate only when the independent
sample reaches ten. The harness also refuses to report precision at all, because the manifest
carries no negative labels and building the denominator by hand would be inventing evidence. It
passes 5 of 5 and exits 0, run again in this pass.
Then the harness broke its own rule in the one place where keeping it cost something. The scorecard printed "20 of 20 manifest offsets verify [100%, n = 20]". Only 10 of those 20 are distinct strings, and 9 are the same boilerplate sentence appearing three times in each of three copies of one document. Nine copies of one sentence are not nine samples, by report.py's own argument. Every metric obeyed the floor except the one where obeying it
would have suppressed a 100 per cent. Fixed as a class rather than at the call site: the independent
sample count is now derived by de-duplicating on the identity that makes two probes the same
evidence, so a caller cannot pass the inflated number (35d82b9). The raw count still
shows — twenty of twenty verifying is a real result, it is just not a rate over twenty samples. |
| Analytics screen | Me, against the demo | Would have built one on request; it is the kind of screen a tool produces happily. | Cut, with the reason written down. An analytics screen for a regulatory analyst is a feature engineers want, and the prep guide names that exact trap. It serves no pain we identified. When the review centre later needed counts, the compromise was recorded too: analytics reports raw counts and refuses percentages under ten items, the same rule the eval harness now enforces in code. |
| Breadth versus depth, after seeing what AI Fund demoed | Me (working notes, then 5fa9fc4) |
Projects, threads, work plans, scheduled runs, knowledge, review centre — the demoed surface. | Accepted the breadth, added the rule that makes it safe. Every surface
that synthesises or counts must display its own coverage. compose_take() refuses to
compose a take that does not record how many findings were withheld for failing verification, and
the withheld count renders at equal visual weight to the count it rests on. The
cost of that breadth showed up as a class, not as a bug: the two widest modules it produced
were the two the product did not serve and no test covered. Both are mounted now
(94ef20f) and neither has a test file of its own;
review_centre.py is the lowest-covered module in the repository at 74%. The recurring
failure is built and not connected. Clarke was built and
never included in base.html. Its engine shipped under a different name than the view
looked for. Nothing loaded .env. The approval route was seeded by an agent that died.
No test caught any of them, because a test verifies capability and says nothing about whether a
thing is wired or seeded. |
| Withheld claims in the UI | Me: an unverified claim must not be a greyed-out assertion | A withheld claim carried its statement into the template, styled as disabled. | Changed the data, not the stylesheet. Withheld claims now render from a dataclass with no statement field, so a template cannot show one even by mistake. The test asserts the fabricated statement string is absent from the response body entirely — not greyed out, absent — because "it looked disabled" is not a guarantee and a stylesheet change could otherwise reveal it. |
| Confidence threshold (ADR-006) | Me | The mechanism: MIN_CONFIDENCE_BP, basis points, read from the environment. |
Shipped at zero, deliberately. Nothing is withheld for confidence alone
today, because there is no evidence for where the line sits and a number picked to look rigorous is
worse than an honest zero. The mechanism is present so that setting it is configuration rather than
a code change, which is what ADR-006 actually requires. The comment in
app/state/claims.py says all of this at the constant. |
| Secrets | Me | Offered to copy the peer project's .env, which would have worked. |
Rejected on width, and the gitignore rule landed first. Five values
copied, not fifty. The ignore rule was committed before the file existed
(7086cf2) because this repository is published with history intact, and a secret
committed here cannot be withdrawn by deleting it later.
The example file then documented the wrong thing for a day. It listed those same five names while the code read fifteen — the database URL, the confidence floor, the approval mode, the invitation switches, the demo-account panel, the cookie flag, the tenancy defaults and five SMTP settings, none of them named. VERBATIM_DATABASE_URL was the
absent one that cost an hour: it is a SQLAlchemy URL, not a path, and setting a path does not fail —
SQLAlchemy quietly opens ./verbatim.db relative to the working directory. Every name in
the file now comes from grep over the code rather than from memory, and the four Gmail
values sit at the foot under a heading saying the application does not read them
(1343f57). |
| A real corpus, and what it may not replace (ADR-40) | Me: the demonstration runs on a filing somebody actually made | 102 public filings retrieved from eight commissions, each with provenance; then the Kentucky
2025-00113 pair loaded into the demonstration workspace (scripts/ingest_real.py). |
Accepted, and the synthetic corpus explicitly kept. The tempting move
is to retire the invented corpus once real filings exist. That would have gutted the eval harness in
silence: its traps are built on purpose — one sentence repeated three times per version, a
restructure scoring 0.944 on raw similarity — and a real filing will not reproduce them on demand.
So synthetic stays the eval corpus, real becomes the demonstration corpus, and both are labelled.
The real pair is the stronger argument anyway, because it is arithmetic rather than a claim:
1,024,409 characters against 1,024,536, a 127-character difference across a million characters, 144
changes found in 0.78 seconds. And the filer published their own marked-up copy, which is the first
answer key on this project nobody on this side wrote. The instruction the agents
were given is the part I would defend: the target of ten filings was an ambition and the
anti-fabrication rule was a floor — an empty data/real/ would be a successful run, a
fabricated one would end the project. |
| Role templates | Me | Ten templates in scripts/seed_roles.py, each named for a function and each drawn
from a title already in the repository. |
The list I handed over was titles; what shipped was functions, and functions
are right. The note I left (docs/.ai/resume.md) said three fixed roles were too
few and gave the real titles in data/real/ as the evidence: Analyst II, Manager,
Director, Deputy Director, VP Pricing & Planning, Regulatory Counsel, Legal Assistant. Half of
those are ranks, not permissions, and taking the list at face value would have built the vocabulary
out of them. A Director of Load Forecasting and a Director of System Operations
approve entirely different things, so a route step reading "Director" says nothing a product can act
on. The templates are functions instead, every one traced to
data/company_context.json or to a filer in data/real/, and all fifteen
permission codes checked against PERMISSION_CODES so a template cannot name a
permission the product does not define. No template holds both action.propose and
action.approve — not because a four-person team may not, but because a default must not
hand it over in silence (03abd37). |
| Documents | Me: the ADRs. The AI: the prose. | Seven documents written from the ADRs — PRD, TDD, architecture, security, MRD, roadmap, README. | Attacked by three adversarial reviews, and they found real things. Two documents answered the moat question in opposite directions: the MRD claimed the verifier and diff engine were the defensible advantage, the roadmap said a competent team rebuilds both in weeks. That is the question every partner asks and the submission had two answers. The verifier is now stated as the floor — necessary, table stakes — and the durable advantage is the accumulated obligation graph, with an explicit admission that it does not exist at launch. |
| The handoff note at the quota wall | Me: stop at a coherent tree, not at a finish line | docs/.ai/resume.md, written to make the restart cheap — and it carried a material
error about how the workflow cache works. |
Corrected minutes later, in its own commit (c7d6e00), rather than
edited quietly. The note first implied completed agents could be replayed from any session.
They cannot: the cache is same-session only, so returning to the session recovers roughly 3.3M
subagent tokens of finished work and starting a fresh one discards all of it. A handoff note whose
whole purpose is a cheap restart is the worst place to be approximately right. |
| Outreach to real people | Me: real names, from public filings only | A drafted outreach batch containing four fabricated facts about real, named people. | Nothing went out until every claim was checked against its filing. The worst entry on this page. See below. Where it stands: 7 messages sent on 2026-08-03 and 38 more sending through 2026-08-04, one per organisation. Zero replies. Zero interviews. Messages sent is an input, not a result, and counting it as progress on user empathy is the exact move the rubric punishes. |
| The document-tree parser, and the decision to withdraw it (2026-08-11) | Me. The tool proposed the parser and I asked for the measurement before the wiring. | A working parser for a filing's own numbering, wired into _alignment_confidence, plus
a measurement harness over the eight real version pairs. |
Rejected the feature and kept the measurement. The numbers said the two branches it was built for never fired, and that every confidence it moved moved because the parser had failed to read a document. A model reading that table would have reported "112 confidences improved". A parse refusal raising a score is absence licensing certainty, which is the one thing this product forbids by name, and the first crossed pair proved it: a gutter refusal on one filing stripped the cap off the other filing's genuine numbered paragraphs. The wiring came out; the parser and the page stayed. The rule I would state at a panel: measure before wiring, and be willing to publish the measurement that kills your own feature. |
| The context budget for a feature that does not exist (2026-08-11) | Me. The tool's first proposal was multi-turn chat. | A priority-ordered budget with a floor that is never evicted, a refusal when the floor will not fit,
and an AST tripwire over app/. |
Rejected the feature and kept the rule. Multi-turn is a real decision and inventing it inside a shim would be smuggling in behaviour nobody chose. But the rule had to be written first, because the policy everyone reaches for — a sliding window over recent turns — evicts the citation offsets that make a claim checkable, and that is this product's central failure arriving through a door marked "context management". Also rejected: reporting a token count. There is no offline tokenizer in these dependencies, so a token figure would be a ratio somebody made up; it is measured and announced in characters, and the measure and the unit have to agree in both directions or it refuses to run. |
| The model-path eval (2026-08-11) | Me, on the shape. The tool wrote the harness. | A scorecard over judge_materiality with a golden set and thresholds. |
Rejected one score and split it in two. A single accuracy number would
report a model that fabricated ten citations and a gate that caught all ten as a failure, when it is the
design working. Fabrication is counted as a property of the model and again as whether any survived the
gate, which must be zero; a miss is counted apart because it costs an afternoon and blocks nothing. Also
rejected: making it a sixth metric in make eval, which would have turned that harness's
"no network call" caveat into a lie the first time it ran. |
| Three rounds of adversarial review over the scalability page (2026-08-11) | Me, in asking for the review. The findings were the reviewers'. | Re-runs of the benchmark that reproduced every structural figure and contradicted two published ones. | Accepted both corrections and republished the worse numbers. A worker was sized from a constant measured over a narrower region than the sentence claimed, erring low — the direction that admits a pair the worker cannot hold. Then the correction itself was traced with the document built outside the region. Every pipeline figure was re-measured and the page got worse: headroom on the admission check fell from thirty per cent to eleven. What I would not accept was a threshold guard on the bytes, because moving the join costs about half a byte per character and the region has more slack than that, so any threshold would be a number tuned to one run. The guard watches where the allocation happens instead. |
The wrong hypothesis is the interesting column. Twenty-three rows, the first of which is open at
the time of writing. Thirteen of them are logged with
their guards in docs/.ai/findings.html; the rest appear here first. They are ordered by
what they cost, not by when they happened.
| Symptom | First hypothesis (and why it was wrong) | What the evidence showed | Fix |
|---|---|---|---|
Open, and red right now. tests/test_invites.py::test_an_invitation_cannot_land_work_that_rests_on_a_proposed_mapping passed at 08:5x UTC on 2026-08-11 and failed at 09:19 UTC. Nothing was committed between the two runs. |
Something in the documentation pass broke it — the only edits between the two runs were prose and a Makefile target. Wrong, and checkable in ten seconds: git stash and the failure stands on a clean tree. |
A clock. The test pins T0 = 2026-08-04 09:00 UTC and mints an invitation with the default seven-day life, so it expired at 2026-08-11 09:00:00 UTC — nineteen minutes before the failing run. Its last assertion calls app/state/routing.py::shared_queue, which takes no now and re-derives every reason against the real clock, so the queue answered ROUTE_OWNER_INACTIVE where the test expects ROUTE_MAPPING_UNCONFIRMED. Every other call in that test is given T0. |
Not fixed. The product is right — a queue that did not say what is true now would be the bug — so the fix belongs in the test, or in giving shared_queue the injected clock every neighbouring function already takes. Two things worth saying about it. It is the same class as tests/test_clock_pinned.py, which exists precisely to stop a real clock reaching test code and did not cover this path. And the failure is dated: this suite carried a test that would go red on a particular Tuesday morning and nothing announced that, which is a sharper version of "prose goes stale and nothing catches it" — a fixture goes stale the same way. |
Fixed; the suite passes on a fresh clone, 2,353 as of 2026-08-08. make test had failed on a
fresh clone of HEAD, the only fatal-class bug in this repository:
tests/test_retention.py::test_every_table_in_the_schema_has_a_window_and_a_reason
failed. 1 failed, 1,380 passed. make fresh-check exited non-zero. |
That a green working copy means a green clone, and that the failures in the working copy came from uncommitted files. Half right, which is why it nearly passed unnoticed: the working copy fails three tests and two of those three are the uncommitted scheduler. The third is not. | Running make fresh-check, which clones HEAD into a temp directory and runs the
suite there. app/state/models.py at HEAD declares a
user_permissions table; app/state/retention.py's
SCHEDULE does not cover it, and the guard's own message says why that matters — "a
table with no window is a table nobody decided about, and the privacy page would be describing a
schedule with a hole in it." A feature went in half-committed: the model
landed and the module that uses it did not. The guard worked exactly as designed and the commit went
in over it. |
Not a documentation fix. Either give user_permissions a
retention window and a reason, or take the table back out until the feature it belongs to lands
whole. The Makefile's own comment says make run breaking is "the only bug in this
repository that is fatal rather than embarrassing"; make test failing is the same
sentence, because AI Fund runs both before booking a panel. |
The landing page links to a screen the product does not serve. Every page's nav
carries "Projects" as a live link to /projects. On the assembled application
/projects is a 404. |
That the unmounted-router defect had been fixed. It had — for the three routers that existed
when the fix was written. tests/test_app_wiring.py was added for exactly this failure
and passes. |
Starting app.main:app and requesting each route.
app/main.py includes proceedings, changes and
review. app/web/views/projects.py (872 lines, five routes) and
app/web/views/review_centre.py (1,149 lines, four routes) are included nowhere.
test_app_wiring.py asserts five paths it names by hand, and neither module's
paths are among them — so the test written to catch an unmounted router cannot catch the next
one. Worse, base.html hardcodes true for the Projects
entry's reachable flag, in a template whose own comment three lines above says screens
without a target "render as plain text, not as links that would 404. Absence is denial here too." |
Closed at 94ef20f. Mounting was blocked by a real
collision: projects.py and proceedings.py both claimed /, and
review_centre.py and review.py both claimed /review. Parallel
agents each assumed they would own the path and nobody reconciled them, so a naive mount would have
shadowed a screen rather than added one. Each router now owns its own: the project list takes the
landing screen, proceedings moves to /proceedings, the review centre keeps
/review, the escalation queue moves to /escalations. And
the class was fixed, not the line. Three assertions derive their answer instead of listing
it: every router in app/web/views must be mounted, no two routers may claim one path,
and every nav link the masthead renders must resolve. The reachable flag for Projects in
base.html is still the literal true — what changed is that a literal which
lies now fails test_every_nav_link_the_masthead_renders_resolves, which fetches every
href the nav prints and refuses a 404. |
Every model path in the product announced itself as unavailable while a valid API key
sat in .env. Clarke answered every question with a polite "the part of me
that reads the record is not wired in". The proposer declined. Mail declined. |
That the key was missing or wrong, because three independent modules all said they had no key. Three agreeing witnesses read as corroboration. | Nothing loaded .env into the process. The value sat in the file and never reached
os.environ, so all three fell back correctly on a condition that was not true. This
is the worst shape a bug can take: every fallback fired, every message was honest, and
nothing looked broken. best-practices §26 says a fallback must announce itself — it did, and what it
announced was false, which is the failure mode §26 does not cover.
Underneath it, a second one of the same family: app/web/views/chat.py resolved its
engine as app.chat.engine.answer while the engine shipped as
app.chat.agent.run_turn. Two agents either side of one seam named it two ways, and a
missing engine is a legitimate state, so nothing failed. |
app/config.py loads the file before anything reads the environment, and
app/main.py calls it above its own imports because
app/auth/policy.py resolves the approval mode at import time. The real environment
always beats the file, and it never logs a value. app/chat/engine.py is the bridge for
the name. (97daf02.) No test covers
app/config.py. |
Caught one redeploy before it happened: every screen on the live site would have
answered (sqlite3.OperationalError) no such column: document_versions.source_url. |
That deploy/entrypoint.sh was safe because it seeded only when there was no
database. That half was right — a redeploy must never lay demo rows over an audit chain. |
When the file existed the script ran nothing, not even
create_all. So the first deploy after a column was added would keep yesterday's table
while today's code selected today's column. Reproduced exactly: a database built from the models
minus the five new columns, then versions_for_company, then the error; then migrate,
then the same read succeeds. No test could have caught it. A test builds
its schema from the current models every time, so it never meets yesterday's database. A subagent
found it by asking what its own change would do to a live host — the question the suite cannot ask. |
app/state/migrate.py, run on every start. Derived, not listed: the obvious fix is
five ALTER TABLE lines, which repairs the instance and leaves the class for whoever
adds the next column. So it asks SQLAlchemy what the models declare, asks the database what it has,
and adds the difference. It never drops, renames, retypes or backfills — a new column is NULL on old
rows, because NULL says the schema of the day did not record this and a default says something the
record cannot support — and it refuses a NOT NULL column with no default and exits non-zero. It also
calls migrate_audit_schema, which had been exported with no production caller at all,
so the live database may have been missing its attribution columns since the day they landed.
(4ac7029.) |
init_db() dropped every table, always, while its docstring said it did so
only for an in-memory test database. |
That the docstring described the code. It had never been checked, and nothing enforced it. | Read during the deployment, not by a test. In this product the default is sharper than usual:
the audit chain is append-only and hash-linked precisely so that no row can be removed unnoticed,
and drop_all removes every row and leaves nothing behind to notice. One call in a
production path destroys the evidence and looks like a clean install. |
The default stays, because 480 test call sites depend on the drop for isolation — but it is
drop_first: bool = True now, stated in the signature, and the docstring carries the
correction rather than the claim. A caller that means "make sure the tables exist" says
drop_first=False and gets exactly that. deploy/entrypoint.sh passes False,
and only when there is no database file. (b5c894b.) |
A share link redirected its own credential. An anonymous open of
/s/<token> answered 303 to
/login?next=%2Fs%2F<token>. |
That the session guard failing closed is always the safe direction. It usually is. | The share prefix was not in the guard's public list, so the guard put a live bearer token into a
query string — where it reaches the access log, the Referer header and browser history.
A redirect that carries the thing it was protecting is worse than no guard,
because it looks like one working. The sharing agent had written an
xfail for this behaviour; it turned XPASS when the fix landed, which is how the fix
proved it hit the right thing. |
SHARE_PATH_PREFIX + "/" joins PUBLIC_PREFIXES in
app/web/deps.py, with the trailing slash load-bearing: /settings and a bare
/s stay guarded, and prefix matching on the rest would make
/login-as-somebody-else public by accident. (97daf02.) |
The test written to catch an unmounted router could not catch the next one.
tests/test_app_wiring.py passed all day while two routers holding 2,021 lines and nine
routes were mounted nowhere and the nav linked to one of them. |
That a regression test named after a defect covers the defect. It covered the instance. | The test asserted five paths it named by hand, and neither module's paths were among them. A list a person maintains cannot catch a module that person forgot. This is the exact failure CLAUDE.md names — fix the class, not the line — committed by the fix that quoted it. | Three assertions that derive their answer from the application: every router in
app/web/views is mounted, no two routers claim one path, every nav link resolves.
(94ef20f.) |
| None. No test failed, and 56 were passing. | That tenant isolation was proved. tests/test_isolation.py existed and passed, so
the chokepoint claim in security.html read as verified. |
Reading the data inventory rather than the test output:
app/diff/engine.py::passage_refs selected passages on version_id alone, and
Passage carries no company_id. Knowing a version id was enough to read
another tenant's source text, and version ids are short and guessable. The isolation test passed
because it never asked this question. |
Join Passage to DocumentVersion and filter on the version's company,
routed through app/state/queries.py so scoping lives in one auditable place.
company_id is keyword-only with no default, so it cannot be omitted or passed
positionally into the wrong slot. Five tests, one asserting the offsets everything depends on survive
the scoped read. (58cfc68, finding #1.) |
Commit 45c408c says the four screens exist and 222 tests pass. Both true. A reviewer
running make run at that commit reached /healthz and nothing else. |
That 24 passing screen tests mean the screens are in the product. | tests/test_screens.py and tests/test_change_view.py assemble their own
FastAPI object around the routers — deliberately, so a screen can be tested before
anything wires it in. app/main.py at that commit called no
include_router. Every test passed because each one built the application the product
did not. |
app/main.py now mounts three routers and the static directory, and
tests/test_app_wiring.py asserts the object make run starts is the one the
screen tests exercise. It also caught a second bug while being written: the tenant was answered
twice, by deps.py and by proceedings.py reading different sources, so a
deployment pointed at a second company had the list stay silent while the change screen served the
first company's source text. The fix was a line, not a class — see the row
above. |
| A parallel run of nine agents produced 39 failures — assertion errors deep inside unrelated modules. | That the new workspace code had broken something badly. The failures looked like real defects and were triaged as real defects. | They moved between runs. tests/conftest.py pointed every process at one fixed
scratch path while init_db() drops tables, so concurrent runs were deleting each other's
schema mid-test. A build agent had flagged exactly this during the engine work.
The warning was read and not acted on. |
The scratch path carries os.getpid(). One integer removes the whole class, and the
conftest docstring now names every way it is reachable — several agents in one tree, pytest-xdist, a
watcher alongside a manual run. (5fa9fc4.) |
| A drafted outreach batch contained four fabricated facts about real, named people. | That the outreach path was low-risk prose and did not need the product's own discipline applied to it. | Checking each claim against the filings before sending: a jurisdiction count borrowed from a different company, a docket attributed to the wrong party, and a client name confused with another. A model wrote them, fluently and plausibly — inside outreach for a product built to catch exactly that. | Nothing went out until every factual claim in every message was checked against the filing it
came from, the same rule the product enforces on itself. Seven messages have since been sent; 38 are
drafted and corrected. The product's answer — verify, and withhold what does not verify — existed
and was not applied here, because the outreach path had no verifier in front of it. It belongs in
findings.html as a twelfth defect and is not there yet. |
| A section restructure scored 0.944 and presented itself as a confident alignment. | That high text similarity means a reliable match — the assumption inside every diff tool. | The corpus was built with Section 6 becoming subsection 5.4 precisely to break this. The words barely move in a renumbering, so similarity peaks exactly when structural identity has changed. | Cap alignment confidence when the leading section label disagrees, so the one case ADR-004 says
must escalate does escalate. (9c3d621, finding #2.) |
A test asserting tzinfo is not None on an audit timestamp failed. |
That DateTime(timezone=True) does what its name says. |
SQLite ignores it. Timestamps came back naive, and a naive timestamp in an audit record cannot be compared across systems — "what did we know, and when" is the only question that table exists to answer. | A UtcDateTime TypeDecorator storing ISO 8601 UTC text, which refuses a naive value
at the boundary rather than storing it and hoping. (9c3d621, finding #3.) |
Caught before it ran: make test would have destroyed the developer's
verbatim.db on every invocation. |
That a test suite is harmless to run. | Self-review of the written plan. app/state/db.py builds its engine at import time
and init_db() drops tables, so any suite importing app. before pointing at
a scratch database destroys the real one. |
VERBATIM_DATABASE_URL is set in tests/conftest.py before any
app. import, with the ordering requirement written into the file so a later editor
cannot undo it by accident. (b70d688, finding #4.) |
/healthz reported the corpus as loaded on a machine where nothing was loaded. |
That a liveness endpoint returning ok tells you the system is working. |
It returned a hardcoded zero and a fixed status — a check reporting what is configured rather than what is there. That is the shape best-practices §26 warns about: a fallback that does not announce itself. | Real row counts, plus a truthful corpus_loaded flag and the exact command to run
when it is false. A missing database answers plainly rather than raising a 500 at a reviewer who has
just cloned the repository. |
security.html described an access control that cannot exist. |
That the control was implemented, because the document said so in the present tense. | An agent ran REVOKE against SQLite and got a syntax error. SQLite has no users, no
roles and no GRANT. The sentence was only ever true of a Postgres deployment. |
Corrected in place with the retraction left visible, and the actual enforcement named: a
before_flush guard for application code, a SHA-256 chain for out-of-band rewrites.
Deleting the sentence would have hidden the lesson. (58cfc68, finding #9.) |
| An outreach address recorded as verified bounced conceptually: two public filings gave two different addresses for the same person. | That "verified against a public filing" is a permanent property of a fact. | A July 2021 certificate of service gives counsel as tnyhart@btlaw.com — sworn,
verified, correct at the time. The April 2026 filing in the same docket family gives
tnyhart@taftlaw.com. Same person, same role, different firm. Nothing announced the
change, and nothing on our side threw. |
Corrected, and generalised into best-practices §28: a verified fact has a shelf life, so bind the
verdict to the version. This is now the real-world argument behind ADR-004 and ADR-005 scoping every
citation to a proceeding version rather than to a proceeding. (3fccb50,
finding #11.) |
docs/.ai/state.json — the one file in the project that "cannot be stale" — went
stale, was recorded as stale, and went stale again. |
First time: that generating a fact from the repository is enough to keep it true. Second time: that naming the fix in writing is close enough to making it. | At c7d6e00 the file reports HEAD 45c408c, 16 commits and 284 tests.
The repository has 19 commits and 296 tests. It lists app/evals/run.py at nine lines;
the file is 196 lines and the harness passes 5 of 5 metrics. It lists evals under
designed_not_built. Every one of those was true when make status last ran. |
Regenerated at 2462561, and stale again as this is written —
the third occurrence. It reports HEAD 2b10bb6 and 39 commits against a
repository at 1343f57 and 43. A fix that depends on a human remembering is not a fix,
it is a note; the class fix is a hook that regenerates on commit, and it is still not built.
And the instrument itself lied. scripts/status.py
matched pytest's output for "N passed" and fell back to 0. A collection error prints no
summary line — one half-written module importing something not yet created aborts the run before a
test executes — so "I could not measure" was written into the file as "0 tests pass", in the one
file ADR-42 makes authoritative over all prose. That zero is indistinguishable from an empty suite
and from a suite where everything failed. It now reports no count, names the module that broke
collection, and says so on stderr (295c40d). Absence is denial applies to our own
instruments. The file today reports "measured": false with the reason, which is the
behaviour working. |
docs/.ai/findings.html marks defects #5 to #8 open. The code closes all
four. |
That a findings log written as the work happened stays current by construction. | At c7d6e00: _spans_of runs on a normalized projection with an index map
(#5), NFKC is skipped for <super>, <sub> and
<fraction> decompositions (#6), soft hyphens are deleted and hyphenated line
breaks rewritten (#7), and verify_citation_for_version() loads the version the citation
names instead of trusting the caller (#8). tests/test_normalize.py went from 6 tests to
18. The log did not move. |
Closed at edad6d3. Each row now names the test functions
that guard it rather than saying "covered", so a reader can go and run them. Two more defects, found
by fuzzing code that already looked finished, got rows of their own at the same time: a collapsed
space claiming an empty raw span, and a quote match reporting a span wider than the quote — which
shifted the occurrence index of every real occurrence after it, so a claim that stated its
occurrence correctly would be refused. The log holds 13 findings now. It had lagged in the safe
direction — understating what works — which is why it survived a full day unnoticed. |
| The eval harness is roughly 1,200 lines of code that produces the numbers, and nothing tests it. | That a module whose output a human reads does not need its own guard. | No test file imports app.evals. make test never runs it, so the only
signal that make eval still works is somebody typing make eval. The same
gap covers app/web/views/projects.py and app/web/views/review_centre.py,
and projects.py's own docstring cites tests/test_project_views.py — a file
that does not exist. A docstring asserting a guard that is not there is the product's failure mode
in miniature. |
Closed at ac6a6fd. tests/test_evals.py, 37
tests. The rule pinned first is the one that refuses to print a rate below ten samples, because a
harness quietly reporting a percentage off six items is worse than no harness — and within hours
that guard was earning its place, when the scorecard was caught printing 100% over 20 offsets of
which 9 are one repeated sentence. make test still does not run make eval,
so the two commands remain separate gates. |
README.md states that app/, tests/ and data/
are empty directories, and that make run and make test are
unimplemented. |
That a file whose own stated rule is "say only what is true at each commit" would be updated at each commit. | All three directories are full and both targets work. The claim was true at
634d559 and has been false since dcb04af. It is the first file a reviewer
opens, and it currently understates the build by the entire build. |
Closed at edad6d3 and again at 75fb92d.
The first fix corrected the empty-directory sentence.
The second found three more claims that had gone false in the same safe direction: the
README's own "what the names promise and the code does not" list still said no model runs anywhere,
rollback does not exist, and reviewer routing is a shared queue. All three had since been built —
app/chat/agent.py and app/interpretation/propose.py call the model,
app/state/rollback.py exports revert_event,
app/state/routing.py routes to an obligation owner. A README that understates is still
a README that is wrong, and a reviewer who checks one claim and finds it stale stops believing the
others. It now carries its line count with the command that produced it. Recomputed here at HEAD
rather than copied from the README —
git archive HEAD | tar -x -C /tmp/… then
find app -name '*.py' | xargs wc -l — the figures hold: 41,771 lines of application
across 69 modules, not the 40 the README claimed, and 30,295 lines of tests. |
| Four workflows died mid-run when the session quota hit. | That a long parallel run either finishes or fails cleanly. | It stops wherever it is. What survived was decided by which agent had written last, not by what was coherent. | Stop at a coherent tree rather than a few minutes later: 068503c was taken with 284
tests passing and docs/.ai/resume.md recording what landed, what did not, the four
resumable run ids and the order to restart them — close-findings first, because it is
the only one that touches neither app/state/models.py nor app/seed.py and
so cannot collide. The note also records the seams the agents reported about their own work, so they
are read rather than rediscovered. |
Five defects share a shape. 56 tests did not catch an unscoped tenant read. 222 tests did not catch that the product served none of its screens. 413 tests did not catch that the product's own navigation linked to a 404. 1,256 tests did not catch that no model path could ever reach the key on disk. And no number of tests could have caught the deploy that ran no migration, because a test builds its schema from today's models and never meets yesterday's database. Every test passed each time and no test asked the question.
The third instance had a fix for the second already in place.
tests/test_app_wiring.py existed because of the unmounted router, and it
missed the next unmounted router — it asserted a hand-written list of five paths rather than deriving
the question from app.routes. CLAUDE.md says fix the class, not the line. That was a
line fix wearing a class fix's name, and the difference stayed invisible for a day because the suite
was green. It is derived now (94ef20f), and the general rule it produced is the one I
would take to the next project: a guard that a person maintains by hand is a guard against
the failure you already had.
The class the suite still cannot reach is prose. Every document correction on this page was found
by a person reading, not by a red test. tests/test_app_wiring.py derives from
app.routes; there is no equivalent that derives a claim in a document from the code it
describes, and the marketing site said every filing was invented for several hours after 102 real
ones landed.
The shared test-database path was flagged by a build agent during the engine work, in writing, before it caused anything. It was read and deferred. It later produced 39 failures that were triaged as real defects during a nine-agent run. The fix was one integer. The lesson is not "listen to the tool" — it is that a warning about a class of failure, filed when nothing is broken, is the cheapest information available and the easiest to discount.
The same pattern repeated on paper, twice. This page recorded state.json as stale and
named the fix as one command. Nobody ran the command, and eight hours later the same file was stale
by three commits, twelve tests and a whole subsystem. It was regenerated at
2462561 and, as this line is written, is four commits behind again. Three occurrences
of the same defect in one file, each time with the fix known and written down. A fix that depends on
a person remembering is not a fix.
The third instance of the pattern cost the most and is the newest.
make fresh-check has sat in the Makefile since the first day, next to a comment calling
a broken make run "the only bug in this repository that is fatal rather than
embarrassing". It was carried as "not verified" for two days, run for the first time in this pass,
and it failed.
Four fabricated facts about real people, written by a model, inside messages selling a system whose entire thesis is that a claim must not assert itself without evidence. The batch was checked line by line against the filings before anything went out, which is the only thing that makes it a story rather than an incident. A verification gate protects what passes through it and nothing else. Verbatim verifies claims about a docket. It had no opinion about claims in an email, and neither did anyone else until the messages were checked.
A security document describing a control that cannot exist on SQLite. A README describing empty directories that are full, and then a README still listing three capabilities as unbuilt after all three were built. An address verified in 2021 and wrong by 2026. A findings log calling four defects open after the code closed them. A privacy page and a subprocessors page telling a reader that every document in the demonstration is invented, hours after 102 real public filings landed. A panel brief rehearsing scripts about a model that did not run. An example environment file naming five variables where the code reads fifteen. All were written in good faith by people describing what they intended to build — which is exactly how a model produces a confident wrong answer. Each correction stays visible with its reasoning rather than being deleted.
Most of them understate the build rather than overstating it. That is the safer direction and it is still a contradiction, and it is why they survived: nothing hurts when a document is modest, so nobody looks.
The sharpest ones are not in documents at all — they are in docstrings.
init_db() said it dropped tables only for an in-memory test database and dropped them
always. _spans_of said "O(n), not O(n·m)" and was measured at 1.29 seconds on a
40,000-character document. normalize() named "the three tags where folding rewrites a
number" and was wrong by 210 characters. projects.py cited a test file that does not
exist. A docstring is the claim a reader is least likely to check and most likely to trust, and this
product exists to argue that an unchecked claim is worth nothing.
Re-executed on 2026-08-05 at commit bd21a5a. gh repo view returns PUBLIC and an anonymous file blob answers 200. The deployed instance answers corpus_loaded: true with four live escalations, so the refusal is visible in the running product and not only in the tests. Five documents fetched anonymously answer 200. One row is still NO and no command changes it: no user has been interviewed.
Two rows re-run on 2026-08-08 at commit ba5965b, and only those two. git rev-list --count HEAD gives 135 commits, none squashed. make fresh-check clones into a temporary directory and passes, in 307 seconds, exit zero. Everything in the paragraph above still carries the 2026-08-05 date, because it was not checked again; a stamp covering claims nobody re-ran is the defect this split exists to avoid.
Three rows re-run on 2026-08-11 on the integration branch, and only those three. make test in the working copy: 2,785 passed, 1 xfailed in 350.73 seconds, exit zero, agreeing with docs/.ai/state.json. make run installs, seeds and serves: /healthz, /login, /explain, /changes/CHG-v1-v2-003 and /actions all answer on this machine. git rev-list --count HEAD gives 183. make fresh-check has NOT been re-run since the five branches landed, so every fresh-clone claim on this page is dated 2026-08-08 and covers a suite 303 tests smaller. Say that rather than letting today's count sit in an older run's sentence.
Status as checked against the repository at commit 1343f57, 97 commits, except where a row carries a later date.
| Check | Status |
|---|---|
make run works from a fresh clone in a clean directory, not just
your working copy | The clone and the install work; the serve step was not
watched. make fresh-check was run at this commit for the first time: it clones
HEAD into a temp directory, builds the virtualenv, installs from
requirements.txt and runs the suite there — all of which succeeded. It does not start
the server, so make run from a clone remains unproven. The application was started here
and serves every screen the nav prints; verbatim.citelocal.ai serves the same object over the
internet. |
make test works from that same fresh clone, and the output is legible to a
stranger | YES. make fresh-check clones the repository
into a temporary directory and runs the suite there, so nothing uncommitted can make it pass: it
passed in 307 seconds, exit zero, and printed
fresh clone: tests pass — on 2026-08-08 at commit ba5965b, over a
suite that was 2,427 tests then. The working copy at 2026-08-11 runs
2,785 passed, 1 xfailed in 350.73 seconds. The fresh clone has not
been re-run since.
It did not always. tests/test_retention.py::test_every_table_in_the_schema_has_a_window_and_a_reason
failed here because user_permissions had been committed as a table while the module
that gives it a retention window had not — a feature landed half. The window is now declared
at app/state/retention.py:585
and that test passes on its own in 0.15 seconds. The break log keeps the original entry, because
the failure was real and a fresh-clone check is exactly what catches a half-landed feature. |
make eval produces numbers a reviewer can read | Yes.
Run again in this pass: 5 of 5 metrics pass, exit 0, no model and no network. It prints counts rather
than rates below a sample of ten and states what the numbers cannot support. It now has tests of its
own — tests/test_evals.py, 37 — and make test still does not run it. |
| Both promised commands are checked by something other than the author's laptop | Yes, as of 2026-08-11, and no earlier. .github/workflows/ci.yml runs make test and make eval on every push and pull request — the documented commands rather than a hand-rolled pipeline, because a bespoke pytest -q stays green while make run is broken. A third step runs the model harness with nothing sent and requires the exit code meaning the model was not evaluated. Two limits: nobody has timed the job on a runner, so the thirty-minute timeout is a guess and the file says so; and nothing deploys from CI. |
| Repository is public, or reviewer access is granted, BEFORE the AI interview begins | YES — public, confirmed rather than assumed.
gh repo view jsahasi/verbatim --json visibility returns PUBLIC and an
anonymous fetch of a file blob answers 200, which is what a reviewer holding no account gets. |
| Commit history intact and not squashed — they read it | Yes. 43
commits, each message carrying its reasoning, its reproduction and what it cost. One note is recorded
rather than rewritten: the three corpus files were swept into d614f7d by a broad
git add while a docs agent was committing, and dcb04af says so. |
| PRD and prototype tell the same story; no capability promised that is not built | Much closer than it was, and not yet true. The README, the
findings log, the ADRs, the briefing and the marketing pages were all reconciled against the code on
the last day. What still disagrees: docs/.ai/state.json reports HEAD
2b10bb6 and 39 commits against 43, and requirements.txt names one importer
of anthropic where there are two. Both understate, which is the safer direction and is
exactly why nobody looked. |
| User-interview findings folded into the PRD, with what changed as a result | No — and this is the exposure. Zero interviews. 7 messages
sent on 2026-08-03 and 38 more through 2026-08-04, one per organisation, every factual claim checked
against its filing first. Zero replies. Four bounces. This is a full quarter of the
rubric and no further engineering recovers it.
The aim was right and the channel was wrong. Every address was read off a certificate of service in a real docket — nothing bought, nothing scraped. The population is checkable in this repository rather than from memory: data/real/ holds 102 filings across 19 dockets and eight state commissions,
naming 48 distinct filing organisations — 26 utilities or their outside counsel, 14 commission
staff, 5 consumer advocates, 2 large-load customers intervening on rates, 1 other intervenor. Those
are the right organisations, reached at addresses they had themselves put on a public filing. Forty
five well-aimed messages produced nothing, so the lesson is about the channel: cold email from an
address they have never seen, inside a two-day window. One introduction through somebody they
already know is worth more than another forty-five. No recipient's name or address appears anywhere
in this repository; docs/user-research.html carries the breakdown and says plainly
which part of it cannot be reconstructed. docs/synthetic-user.html exists and is labelled at the
top as not evidence; if a sentence from it ever reaches the PRD as a finding, the submission has done
itself active harm. |
| Citation verification is visible in the running product, not only in the code | Yes, on the change screen, and now on a real filing. The
paired demonstration works from make run: one claim whose citation verifies and renders
with an openable source span, one whose quote was corrupted and never renders as a claim at all. The
review centre, which shows the same discipline across a whole project, is served since
94ef20f. The landing page holds thirty seconds of the running product, recorded with
playwright against the seeded corpus, whose longest beat is the withheld claim — the source says 20
megawatts, the citation quoted 10, and where the statement would be there is a hatched panel reading
no claim made. The poster frame is that panel, so a reader who never presses play has still seen the
argument. |
| Every doc link is viewable by anyone with the link | YES. Checked on 2026-08-05: five documents fetched anonymously from verbatim.citelocal.ai/docs/ — prd, tdd, submission, tech-questions-faq and decisions — all answered 200. |
| 90-second pitch said out loud, timed | Written in
docs/.ai/briefing.html. Not timed aloud on the record. |
| DeepInterview started within 30 minutes of sending the submission, repo open alongside | Not yet. Plan submission and interview as one sitting; the assignment expires 72 hours after the challenge email. |
Saying "I don't know" is described in the prep guide as a positive signal. These are ordered by how much they matter.
text-embedding-3-small
— 145 chunks, two requests, 51,854 tokens, about a tenth of a cent — and the width came back 1536,
not the 1024 the code had named. The comment beside the request said the endpoint would refuse an
unknown field with an HTTP 400; it answers 200 and ignores the field, which is the worse outcome,
because the request would have carried a word that reads like a parameter, changed nothing and passed
every check forever. That is this product's own failure mode found in this product's own source, and
only by making the call instead of describing it. What is still untested against the wire is
the refusal path: no bad key and no wrong model name have been put to the endpoint, so both
error branches are still written from documentation. Nothing is known about load, retries or longer
inputs either. One call proves the wire format, the auth and the parse, once._similarity took difflib's autojunk default, so on any passage over 200 characters most of the alphabet was discarded and the score was computed over what survived. It could only push a score down, never up, so it over-flagged and could not under-flag — the cost was not a wrong approval but a caution a reviewer learns to skim, which is the flag being worth nothing on the day it is right. The keyword was never the work: the value is stored on every change row, so changing the computation alone would have left the screen, the database and docs/scalability.html each reporting a different number for the same change. scripts/remeasure_alignment.py is the derived-corpus migration playbook §§27 requires, and it ran: 333 modified changes read, 25 moved, 2 cautions lifted, 0 added, and a second run moves nothing. The zero is the part worth saying out loud — ADR-97 argued the defect could only push a score down, and 333 real rows agreed rather than the argument agreeing with itself. ADR-97 carries the decision and a correction to its own measurement.docs/.ai/state.json, and that file
had gone stale in the same direction as the prose. A checker that reads one stale source cannot see
agreement as a fault. It is fixed by regenerating the file from a real run rather than by editing the
prose, and the fix deliberately left one guard red for a commit rather than publishing a number that
was not measured.| Gap | Why it is this way | What I would do with more time |
|---|---|---|
The repository is public and the fresh-clone suite passes. When this entry
was written the repository was private and make test failed one test on a fresh clone
of HEAD. | A feature landed half-committed — the
user_permissions table is in app/state/models.py and the module that uses
it is not — so a derived guard in tests/test_retention.py caught exactly what it was
written for and the commit went in over it. The visibility was never changed from the default and
was recorded as unchecked for two days rather than checked in ten seconds. | Both before anything else on this page. Neither is engineering. Details on the checklist and in the first row of the break log. |
| Zero user interviews. ADR-001's persona is a hypothesis, and the PRD says so. | Outreach was drafted on day one and held: the first batch had to be rewritten after four fabricated facts were found in it, and regulatory affairs is a small connected world where a bad first message is hard to undo. 7 messages went out on 2026-08-03 and 38 more through 2026-08-04, one per organisation. Zero replies. Four addresses bounced, and every one of them was filing-verified — a service list records who was reachable when it was filed, not now, which is best-practices §28 arriving in the outreach rather than in the code. | Keep to one message per organisation. Twenty minutes each. Report what changed even if the answer is that ADR-001 was wrong — especially then. Two people have written in unprompted and are unanswered; those are the only live paths to a real interview and they are worth more than another wave. |
The two widest view modules have no test file of their own.
projects.py (872 lines) and review_centre.py (1,149 lines) are mounted and
reachable, and review_centre.py is the lowest-covered module in the repository at
74%. projects.py's own docstring cites tests/test_project_views.py, a file
that does not exist. | Both landed in the hours before a quota wall, and being unmounted hid
the absence: a screen nobody serves is a screen nobody misses. Mounting them
(94ef20f) closed the reachability half and left this half open. They are exercised
indirectly by tests/test_app_wiring.py, which signs in and requests every
screen. | A test file each, and delete the docstring line that names a guard which is not there. A docstring asserting a test that does not exist is this product's failure mode in miniature. |
app/config.py and app/state/migrate.py have no tests
at all. Between them they decide whether the process can see its key and whether the live
schema can be read. | migrate.py is genuinely awkward to test: a test builds its
schema from today's models, so it never meets yesterday's database, which is the whole reason the
defect it fixes reached production. config.py has no such excuse — it is fifteen lines
and was written in a hurry. | For config.py, the assertion that matters is that
the real environment beats the file, because reading it the other way round means a container cannot
override its own image. For migrate.py, build an old schema on purpose and migrate it —
the fixture is the work, not the test. |
| "Verified" means the quote is real, not that the interpretation is right. | Offset verification proves the words exist where claimed. It proves nothing about whether the reasoning drawn from them is sound. | Nothing closes this fully. The direction is an adjacency check on governing qualifiers, and a second model asked to argue the opposite reading — both mitigation, neither proof. Volunteer this before being asked. |
A model now runs in exactly two places, and the eval harness measures neither.
Clarke (app/chat/) and the claim proposer
(app/interpretation/propose.py). make eval scores the deterministic spine
and says so at the foot of every run: no model runs there, no network call is made, and a green
scorecard must not be read as though it said anything about model
behaviour. | Deliberate ordering: the deterministic gate had to work first, because it is the part a reviewer can audit without trusting anything about the AI. Neither model path sits on the verification path — a claim survives its citation whatever produced it — so the model can be wrong without the product asserting something false. | Evals for the model path, reported with
variance across repeated trials, which is what ADR-38 asked for and nobody has built.
Change.materiality still stays NULL rather than defaulting to a value that reads like a
judgement. |
The obligation graph exists as tables and not as a join.
Obligation and ChangeObligation are real tables now, seeded from
data/company_context.json with owners resolved to real accounts, and
app/state/routing.py routes a change to the person who owns the obligation it touches.
What does not exist is anything that proposes the mapping: every link is recorded with the
actor who made it, and no code computes one. No retrieval was ever built either — ADR-08 chose BM25
first, and grep -rn embedding app/ finds nothing. | The mapping is the product's core join and the hardest thing on the roadmap to do honestly, so it was ordered behind the deterministic gate. Recording who made a mapping — a person or the pipeline — was built first on purpose, because a mapping a machine proposed and one a person made must not read alike on the screen that routes work off them. | Build the retrieval and the proposer, and keep the author field doing its job. This is where ADR-008 finally earns its place, and it is also where the MRD's durable-advantage claim lives — the accumulated obligation graph, which the MRD already admits does not exist at launch. |
Nothing fetches on a schedule, and retention is written but not
running. A new version of a docket arrives when somebody puts it there. The retention
windows in app/state/retention.py are declared and no code in the product calls
them. | Both were ordered behind the correctness core on purpose, and a scheduler that deletes customer rows or bypasses an approval is the last thing that should start because somebody deployed an image. | A fetcher per commission and a job runner, with the retention sweep in dry mode first and a banner on every start that says which jobs are on — because a silent boot reads as "nothing is scheduled" and as "scheduled and not fired yet", and those are different facts. |
Precision is not reported, and cannot be. make eval prints
recall counts and refuses a precision figure. | The manifest labels five changes, not every change, so the corpus carries no negative labels. Building a denominator by hand from those numbers would be inventing evidence. | Label the corpus exhaustively, or label a second corpus. Or use
what arrived on the last day: three families in data/real/ carry the
filer's own marked-up copy, which is ground truth nobody on this side wrote. That is the
first real answer key this project has had, and it retires the "the eval set is self-labelled"
concession the moment somebody scores against it. Nobody has yet. |
Documents lag the code, though far less than they did. The README, the
findings log, the ADRs, the briefing and the marketing pages were reconciled on the last day.
docs/.ai/state.json is the one still behind — HEAD 2b10bb6 and 39 commits
against 43 — and requirements.txt names one importer of anthropic where
there are two. | Each was true when written. Every one understates the build, which is why
nobody noticed — an overstatement gets caught the first time somebody checks. The guard for the
application is tests/test_app_wiring.py deriving from app.routes. There is
no equivalent for prose, and every correction on this page was found by a person
reading. | Run make status from a commit hook rather than from memory. Beyond
that, the interesting question is what a derived guard for prose would even look like: a claim in a
document that names a file and a symbol could be checked, and most of the false claims on this page
were exactly that shape. |
Authentication is built, and the segregation-of-duties hole in it is real.
A person signs in over the internet. An admin holds user.manage, so an admin can grant
themselves obligation_owner and approve their own work. Separately,
can_approve had no caller in app/web/ for a day — an independent review found that
before I did. It has one now: app/web/views/actions.py::decide, the screen where a proposed
action is approved (ADR-91). The fix was not a line. There was nowhere in the product to propose an
action at all, so the gate had no decision under it, and two attempts to wire it into the escalation
queue were reverted because that screen belongs to the analyst, who does not hold
action.approve. Closing an escalation is still gated on escalation.resolve
alone, so an operator can still resolve an escalation they raised. | The chain records the grant, which makes it
visible afterwards and does not prevent it. Both limits are written in
docs/security.html in those words rather than implied away, because a compliance
product that oversells its own controls has made the argument against
itself. | A second approver on privilege changes, and an approval that
performs the action rather than recording it. The first is design work. The second is the honest
limit of what shipped: /actions records what the company decided and files nothing. |
The hosted instance runs the product now, and one ADR still describes something
else. verbatim.citelocal.ai serves deploy/site/ at the root with the
application behind it, nginx proxying a named list of paths rather than a catch-all. Checked from
here: /healthz answers corpus_loaded: true. | A catch-all fails
quietly in both directions — a page added to the site would silently reach the app, and a router
added to the app would silently 404 — so the list is the seam and it is written down. The app
container carries no Traefik labels and publishes no port, so nginx decides what reaches it.
ADR-09, ADR-10 and ADR-11 were all false and were corrected in place at 295c40d:
Traefik under Coolify serves the site, not the Caddy that ADR-11 chose, and the host is citelocal-1
rather than "its own droplet" — chosen because it is not the box carrying PII or PHI, which honours
the reasoning and breaks the letter. | The local path stays the contract either way, which is
what ADR-009's mitigation promised, and make run is still what a reviewer is asked to
trust. |
Rollback is built (T13); the review queue still has approve and reject, not amend
(T16). app/state/rollback.py exports revert_event, guarded by
tests/test_rollback.py (38). There is no point-in-time replay. | Amend was on the cut list and ADR-013 permitted it explicitly: the module keeps the third path unused rather than the architecture being rewritten around a temporary UI limit. | Amend is an event appended to the log like any other, so the work is the editing surface, not the model. |
| SQLite cannot do row-level security, so tenant isolation is an application-layer promise enforced by tests rather than an engine guarantee. | ADR-007:
a reviewer runs make run before scheduling a panel, and every service they must install
first is a way for the submission to fail on a machine nobody has seen. Finding #1 is exactly what
that choice costs. | Reversal condition is a trigger, not a date: a second tenant, or any real customer data. SQLAlchemy already makes Postgres with RLS a configuration change. |
| The audit hash chain detects tampering; it does not prevent it. | Two independent mechanisms because they fail differently, and both are honest in the module docstring about what they do not do. Anyone who can write the whole file can recompute the chain forward. | Publish the chain head where the same attacker cannot reach it. |
Last verified against the repository at commit 1343f57: 97 commits;
1,380 passed and 1 failed on a fresh clone at that time; 1,378 passed and 3 failed in the working copy; 92% line
coverage as recorded at 2b10bb6; make eval 5 of 5, exit 0; the hosted
instance answering corpus_loaded: true; the repository PRIVATE at that moment. The tree
was not clean at the moment of the check, and that matters for reading the numbers
above: a parallel workstream held uncommitted work — a scheduler under app/jobs/, a
docket fetcher at app/sources/fetch.py, point-in-time replay at
app/state/replay.py, and granular permissions across
app/state/permissions.py and app/web/views/permissions.py, with their test
files. None of it is claimed as shipped anywhere on this page, and the gap
rows that say those capabilities are missing are correct against HEAD, which is what a reviewer
clones. Every claim above is bound to 1343f57 and expires against anything later —
best-practices §28.
Written for five minutes, with a three-minute cut marked. If the brief names a different length, cut from the widening section and never from the cold open. Say it in your own words -- this is scaffolding, not a teleprompter. Where a line sounds like writing rather than speech, it is wrong and you should change it.
Record the screen, not your face, for the first ninety seconds. The reviewer has watched many people introduce themselves. Almost none of them opened on the product doing something.
Screen is already on a change with a claim whose citation does not verify. No title card, no "hi, I'm". First words over the product:
"This is a regulatory filing that changed between two versions. The system found a claim about it, and it is refusing to make that claim. The words it wants to quote are not at those offsets in the source any more, so it will not say the sentence. It says why, and it says who to ask.
Every AI product I have used would have said the sentence."
Then stop talking for two seconds and let them read the screen.
Why this open: the prep guide says the hardest technical decision must be visible in the product rather than buried in the stack. This is that decision, on screen, in the first thirty seconds, before you have asked them for anything.
"This is built for one person. A regulatory affairs analyst at a utility that operates in more than one state. Not a lawyer, not a compliance officer, not an executive — one person.
Their day: a commission posts a revised order. Somewhere in four hundred pages, something moved. They have to find what changed, work out which of their company's obligations it touches, and tell the right owner — before a deadline they did not set. Today that is done by reading both versions side by side.
The moment this helps is narrow and I can name it: they open the revision and the changes are already found, each one carrying the exact words that changed and where."
Say "one person" out loud. The guide is explicit that naming more than one type of user means the work is not finished.
Move through, narrating what matters rather than what you are clicking:
Three-minute cut: keep the citation that verifies, drop the diff and the routing, go straight to 2:30.
"The hard part is not calling a model. The hard part is that a language model will produce a fluent, plausible, wrong sentence and nothing downstream can tell.
So the deterministic half runs with no API key at all. The diff, the citation verifier, the tenant scoping, the audit chain — none of that asks a model anything. The model proposes; it never asserts. A person confirms, and the record says which.
Verification happens on the read path, not the write path. Every render re-reads the source at the cited offsets. That costs real milliseconds and it is the only way the answer stays true after the source moves."
Have one file open behind this if you can. Vague architecture talk is called out by name in the prep guide as a fail signal.
Do not skip this and do not soften it. The guide says "I don't know" is a positive signal and bluffing is a red flag. Pick two or three from the gaps table above and say them plainly. Then:
"The thing I would change: [name it]. The thing I still do not know: [name it]."
Energy here, not a summary. No recap of what they just watched. Say what makes this urgent, what you would build in the next month, and what you would need to find out first. End on the next thing you would do, not on thank you.