Every defect found during the build, how it was found and what was done about it; then every judgement call, with its alternatives and what would reverse it. Written as the work happened, not reconstructed.
Why this file exists. A repository that only shows its finished state teaches a reader nothing about how carefully it was built. The interesting information is in what went wrong and what was decided under pressure. Two patterns in the table below are worth naming up front: most defects were found by adversarial review or by fuzzing, not by a failing test, and three of them were false claims in documents rather than bugs in code — which matters for a product whose entire thesis is that a claim must not assert itself without evidence.
| # | Defect | How found | Why it mattered | Resolution | Guard |
|---|---|---|---|---|---|
| 1 | passage_refs read the passages table with no company scope. Passage carries no company_id. |
Adversarial review of the data inventory. No test failed. | Knowing a version id was enough to read another tenant's source text, and version ids are short and guessable. It also contradicted the docstring in models.py and security.html's claim that every read passes through one chokepoint — the exact doc-versus-code mismatch this project's thesis forbids. |
Fixed. Join Passage to DocumentVersion and filter on the version's company. Routed through app/state/queries.py. company_id is keyword-only with no default so it cannot be omitted or mispositioned. |
tests/test_passage_isolation.py, 5 tests including one asserting offsets survive the scoped read |
| 2 | Diff alignment confidence was raw text similarity. | A test in the plan failed against the plan's own implementation. | Backwards in the dangerous direction. The corpus restructure — Section 6 becoming subsection 5.4 — scored 0.944 and presented itself as a confident match. A renumbering is dangerous because the words barely move, so similarity runs high exactly when structural identity has changed. | Fixed. Confidence is capped at 0.5 when the leading section label disagrees, so the one case ADR-004 says must escalate does escalate. | 3 regression tests, including one proving an ordinary in-section edit keeps its true score rather than escalating everything |
| 3 | Audit timestamps came back naive. | A test asserting tzinfo is not None, written before the implementation. |
SQLite ignores DateTime(timezone=True). A naive timestamp in an audit record cannot be compared across systems, and "what did we know, and when" is the question the table exists to answer. |
Fixed. A UtcDateTime TypeDecorator storing ISO 8601 UTC text, which refuses a naive value at the boundary. |
tests/test_audit.py |
| 4 | conftest.py would have destroyed the developer's verbatim.db on every make test. |
Self-review of the written plan, before any code ran. | app/state/db.py builds its engine at import time and init_db() drops tables. A test suite that damages the thing it tests. |
Fixed before it shipped. The scratch database URL is set before any app. import, with the ordering requirement written into the file so a later editor cannot undo it by accident. |
The ordering is documented in the conftest docstring |
| 5 | _spans_of falls back to fixed-width window scanning. |
Adversarial review during integration design. | It only works while normalization preserves length, and normalization collapses whitespace. On real PDF text the occurrence check silently stops finding spans, which disables the boilerplate-trap guard — one of the product's strongest ideas, failing quietly. | Fixed in bbd44fb. The window scan is gone. normalized_projection() in app/text/normalize.py makes one left-to-right pass over the source and carries, for every normalized character, the raw offsets of the run that produced it. _spans_of searches in normalized space and maps each hit back. O(n), where the scan was O(n·m) and wrong. |
tests/test_pdf_text.py, built PDF-shaped and recording each raw span as the fixture string is assembled, so the expected offsets do not come from the code under test: test_the_spans_found_are_the_raw_offsets_recorded_when_the_fixture_was_built, test_all_three_occurrences_are_found_in_pdf_shaped_text, test_normalization_changes_the_length_of_every_recorded_span (which pins the assumption the old scan made), test_each_occurrence_reports_its_own_index_in_document_order, test_the_fixture_really_carries_all_three_extraction_artefacts. Plus tests/test_normalize.py::test_a_normalized_span_maps_back_to_the_raw_offsets_that_produced_it. |
| 6 | normalize() folds superscripts and fractions via NFKC. |
A build agent reading its own implementation against its docstring. | normalize("20²") returns "202" and "½" becomes "1⁄2". The docstring promises digits and units are never folded. A footnote marker or a fraction in tariff text can change value under normalization. |
Fixed in bbd44fb, by narrowing NFKC rather than the docstring. A character is held back from folding when its compatibility decomposition is tagged <super>, <sub> or <fraction>, or when folding it would begin with a decimal digit that then closes up against the number to its left. A character that is already a digit still folds, because a full-width two is the digit two wearing another face. Amending the docstring was the other option and was rejected: the honest version would have had to say the module rewrites values, in the one module whose job is to preserve them. |
tests/test_normalize.py: test_superscripts_and_subscripts_keep_their_value, test_vulgar_fractions_are_not_expanded, test_still_folds_the_compatibility_forms_that_carry_no_value, test_the_protected_forms_survive_a_second_pass, and test_the_folding_boundary_is_exactly_where_the_docstring_puts_it, which holds both halves of the rule in one test so neither side can drift without the other being read. |
| 7 | normalize() has no rule for soft hyphens or line-break hyphenation. |
Inbound-integration research into PDF extraction. | PDF text carries U+00AD and -\n word breaks. Both defeat exact citation matching, which is the product's foundation. |
Fixed in bbd44fb, as planned. U+00AD is deleted unconditionally, including between a letter and its combining accent, so a second pass cannot compose what the first left apart. Any dash against a line break loses the break and keeps the hyphen, over every character that ends a line — the form feed included, because that is the break a paginated filing is guaranteed to carry. A true word break is not rejoined: "demon-\nstrate" stays "demon-strate" and fails to match, per ADR-003. One limit is conceded rather than closed: a soft hyphen sitting at a line break gives "main tain", which matches nothing and goes to review. Wrong text, and the safe wrong text — joining it would be a guess that fails open. |
tests/test_normalize.py: test_deletes_the_soft_hyphen_without_leaving_a_gap, test_a_soft_hyphen_does_not_strand_an_accent, test_a_hyphen_against_a_line_break_loses_the_break_and_keeps_the_hyphen, test_a_true_word_break_is_not_rejoined, test_a_spaced_dash_is_left_alone, and test_a_soft_hyphen_at_a_line_break_fails_closed_rather_than_rejoining, which pins the conceded limit so it cannot change by accident. End to end in tests/test_pdf_text.py: test_a_hyphenated_word_break_is_not_counted_as_a_fourth_occurrence and test_a_real_hyphen_split_across_a_line_break_still_verifies. |
| 8 | Citation.version_id is carried but never checked against the source text passed in. |
A build agent reporting on the plan it was given. | A citation naming v2, verified against v1's text, passes if the quote matches at those offsets. The repeated boilerplate sentence across versions makes this reachable in this corpus. | Fixed in bbd44fb. verify_citation_for_version(session, citation, company_id, ...) loads the version the citation names, through the tenant chokepoint in app/state/queries.py, and verifies against that text. A version this company cannot read and a version that does not exist return the same refusal, so the answer never tells a caller which ids exist. The two-argument form stays, with its limit written on it, because the pure function is what the diff and the evals score. |
tests/test_verification.py, seven tests where there were none. The pair that makes the hole legible: test_the_two_argument_form_cannot_catch_a_mispaired_version and test_the_version_aware_form_rejects_that_same_citation — the same misfiled citation, verified by one and refused by the other. Then test_the_version_aware_form_verifies_the_citation_when_it_names_its_own_version, test_the_version_aware_form_still_applies_the_occurrence_rule, test_a_version_this_company_cannot_read_is_refused_not_answered, test_the_other_tenant_is_refused_even_when_the_quote_is_correct, test_an_unscoped_read_is_refused_rather_than_treated_as_a_wildcard. |
| 12 | A collapsed space could claim an empty raw span. normalize("A ́B") gave the emitted space the span (1,1). |
A fuzzer, over roughly 60,000 random strings assembled from PDF artefacts, run against code that already looked finished and had a green suite behind it. | The whitespace run was closed at the next chunk's start, which is one character too early when a single chunk folds to a space followed by something else — an ideographic space carrying a stray combining mark, which bad PDF extraction does emit. The Projection docstring promises that every normalized character names the run of source characters that produced it. A reviewer clicking through to see the cited characters would have been shown nothing, and the promise would have been false while every test passed. |
Fixed in bbd44fb. The run ends where the chunk that produced the space ends, not where the next chunk starts. The span is then the whole chunk — wider than strictly necessary, and honest: those are the source characters the space came out of. |
tests/test_normalize.py::test_a_collapsed_space_never_maps_to_an_empty_raw_span, which asserts start < end for every character rather than only for the one that failed. The class is held by test_every_projection_offset_is_a_real_non_empty_run over text mixing every rule in the module. |
| 13 | _spans_of could report a span that is not the quote. |
The same fuzzer, same run. | A needle beginning part-way through an expanded character — the "2" of a squared-metre glyph, which normalizes to "m2" — matches in normalized space with no raw span behind it, because there is no such thing as half a source character. The code answered with the whole glyph, which normalizes to more than the quote. That is worse than a miss: a spurious span shifts the occurrence index of every real occurrence after it, so a claim that stated its occurrence correctly gets refused. The repeated-boilerplate guard would have started rejecting good citations, which is the failure mode an analyst stops trusting fastest. | Fixed in bbd44fb. Every candidate span is re-read and kept only when the raw characters really do normalize to the needle. A cheap check that makes the function's promise exact instead of approximate. |
tests/test_occurrence.py::test_a_quote_beginning_part_way_through_an_expanded_character_reports_nothing, which also asserts the quote taken from the glyph's own start still returns its span, so the fix refuses the wrong span without losing the right one. The promise itself is asserted directly by test_every_span_reported_really_normalizes_to_the_quote. |
| 14 | The documented way to watch the approval gate refuse did not exist. README step 6, ADR-91, the comment block in app/seed.py and the panel brief all said: sign in as the admin, open /users, give the analyst the obligation owner role. No route does that. |
Adversarial review of the change that wired the gate. Every test passed, including the ones that prove the refusal, because they arrange it through identity.grant_role in Python. |
A reviewer following the walkthrough reached a dead end at the exact moment the product's load-bearing control was supposed to appear — and the 403 they did reach was gate 2 (no action.approve), which looks like the same refusal and is not. Worse than a missing feature: a false sentence in four files, in a submission whose thesis is that a claim must not assert itself without evidence. The ninth instance of this project's own pattern, one level up: the control was wired to a screen, and the state that makes it refuse was reachable from nothing. |
Fixed. The sentence was the defect, not the code. Nothing in the product can put a second role on an existing account, and that is the grid working: admin holds user.manage and no approval code, every grant path is ceilinged against the granter, and the one declared waiver refuses an active address. So the four documents now say that, and app/seed.py::ensure_refusal_arrangement — off unless VERBATIM_DEMO_REFUSAL is set — arranges it as the system actor and prints why no screen could. |
tests/test_seeded_refusal.py. The walkthrough is executed end to end through the real POST; a second test asserts that no product route can reach the same state, so if one is ever built the guard goes red and the docs get rewritten rather than drifting; two more hold the four files to the true sentence. |
| 15 | The propose form hid a claim after any proposal, not after a live one. One rejection removed that claim from the form for good. | The same review. {row.claim_id for row in rows} with no state filter. |
On a screen whose argument is that a rejection sends the work back rather than closing it, the analyst could not offer an alternative. The POST accepted the claim the whole time, so the product could do the thing and would not offer it — the version of this defect that is hardest to notice and easiest to mistake for a rule. | Fixed. Only a proposal in STATE_PROPOSED takes its claim off the form. |
tests/test_actions_screen.py::test_a_rejected_proposal_gives_its_claim_back_to_the_form, which rejects through the real route and then proposes again; and test_a_live_proposal_still_keeps_its_claim_out_of_the_form for the half that has to stay true. |
| 16 | decide() caught SQLAlchemyError only, so a second decision arriving between the state check and the write answered 500. |
The same review, by reading the exception paths rather than by racing it. SQLite serialises writers, so it was never reproduced. | record_decision refuses an already-decided row with ValueError, correctly. The handler's own pre-check answers 409 and cannot close the gap between itself and the write. Two callers pressing Approve together would have got a stack trace instead of the conflict the single-threaded path gives. |
Fixed. ValueError answers 409 and LookupError answers 404, so the store's refusals and the view's own reach the same two codes. |
tests/test_actions_screen.py::test_a_decision_taken_between_the_check_and_the_write_answers_409, which makes the store raise at the moment of the write. |
| 17 | app/diff/engine.py::_similarity builds SequenceMatcher(None, a, b) and takes difflib's default, autojunk=True. Here b is a passage's characters, so in any paragraph over 200 characters every character appearing in more than one per cent of the positions is junked — in English prose, most of the alphabet. The alignment confidence on the change screen is then a ratio between what survives. |
Adversarial review of the scaling work, and then measured rather than argued: scripts/bench_diff.py::similarity_accuracy_measurement over 800 real paragraphs from data/real/, 200 in each of four length bands, each changed by exactly one word. |
LOW_ALIGNMENT in app/web/views/changes.py is 0.50. 11 of the 800 paragraphs — one in 73 — cross that line after a one-word edit, and the worst scores 0.444 against a true similarity of 0.964. Each carries a low-alignment caution the text does not support. The direction is safe: junking only removes matches, so the score can only fall, and it fails toward review as ADR-003 requires. What it costs is trust in the caution. A reviewer who keeps seeing the flag on changes that plainly moved one word learns to ignore it, and then it is worth nothing on the day it is right. |
Closed 2026-08-11, and it was open for hours rather than weeks. _similarity now passes autojunk=False and scripts/remeasure_alignment.py carried the reload that made it safe to: 333 modified changes read, 25 moved, 2 cautions lifted, 0 added, largest move 0.253 to 0.830 on a real Kentucky filing, and a second run moves nothing. The zero matters — this row argued the defect could only push a score down, and 333 real rows agreed. ADR-97 carries the decision and the correction to its own figure. What follows is the reasoning as it stood when the row was written.The fix is autojunk=False in _similarity, which the same measurement prices at 9x to 42x on the character path depending on passage length. More to the point it rewrites alignment_confidence on every stored change row, so it is a derived-corpus migration (best-practices §27) and not a one-line edit: until the corpus is reloaded, the screen, the stored rows and docs/scalability.html would each say something different. It needs its own ADR and its own reload. |
None. Every fixture in tests/test_diff.py is under 200 characters, so difflib's cutoff never fires and the ratio is exact — which is why no test can see this. The record is docs/scalability.html section 7 and this row. |
| # | False claim | The truth | Resolution |
|---|---|---|---|
| 9 | security.html: "the database user the application runs as should have no UPDATE or DELETE grant on that table". |
Unavailable on SQLite, which has no users, no roles and no GRANT statement. An agent ran REVOKE against SQLite and got a syntax error. The sentence was only ever true of a Postgres deployment. |
Corrected in place, with the retraction left visible and the actual enforcement named: a before_flush guard for application code, a SHA-256 chain for out-of-band rewrites. |
| 10 | README.md stated data/ was empty, and that app/ and tests/ were too. It also gave make load-corpus as the reload command. |
The corpus landed in commit dcb04af, and the modules and tests over the commits after it. The README was never updated, so it described a repository that had not existed for a day. make load-corpus is not a target and never was; the target is make seed. |
Corrected. The status section now names what is built, what is tested, and the three areas whose name promises more than the code does — no model runs, rollback does not exist, and reviewer routing is a shared queue. make run, make test and make eval were each executed before the sentences describing them were written, because a reviewer runs the commands before reading the prose, so a wrong command there costs more than a wrong sentence anywhere else. |
| 11 | An outreach address recorded as verified: tnyhart@btlaw.com. |
Verified, sworn, and correct — in a July 2021 certificate of service. The April 2026 filing in the same docket family gives tnyhart@taftlaw.com. Same person, same role, different firm; nothing announced the change. |
Corrected, and generalised into best-practices.html principle 28: a verified fact has a shelf life, so bind the verdict to the version. This is now the real-world example behind ADR-004 and ADR-005. |
Each of these was a choice between defensible options, taken under time pressure. Recorded with what would reverse it, because a judgement without a reversal condition is just a preference.
Judgement. Keep SQLite. Do not switch tonight.
Why. ADR-007's reasoning holds: a reviewer runs make run
before scheduling a panel, and every service they must install first is a way for the submission to
fail on a machine nobody has seen. The README calls make run breaking the only bug here
that is fatal rather than embarrassing.
What SQLite genuinely cannot do, stated so the limit is not hidden: row-level
security, so tenant isolation stays an application-layer promise enforced by tests rather than an
engine guarantee — and finding #1 above is exactly what that costs. And revoking
UPDATE/DELETE on the audit table, per finding #9.
Reversal condition. A second tenant, or any real customer data. Not a
date — a trigger. The intended path keeps SQLite as the default so make run stays
dependency-free, and points the hosted instance at Postgres with RLS, which SQLAlchemy already makes
a configuration change rather than a rewrite.
Judgement. verbatim.citelocal.ai is the primary surface; the local
run stays guaranteed. Recorded as ADR-009.
Why it is uncomfortable. The prep guide says reviewers execute both commands before scheduling a panel. Leading with a URL adds uptime, TLS and DNS to the list of things that can lose the submission, none of which existed before.
Mitigation. The local path stays green and is verified on a fresh clone, so the hosted instance can fail without taking the submission with it. If only one survives, the local one is the one that matters.
Judgement. Spend 30–45 minutes and a few dollars a month on a disposable box. Recorded as ADR-010.
Why. The existing host's own runbook keeps it as a single private trust domain because it stores real family PII and PHI. Container isolation would probably have held. It is precisely the assurance that posture declined to rely on, and a day-old application whose URL is going to reviewers is not the thing to test it with.
Judgement. Keep them. Add the provenance note.
Why. Every person named filed something in a public proceeding, and every address came from a certificate of service — a document the parties file, under oath, so that others can reach them about that docket. None came from a contact-data broker. Removing them would leave the outreach claims unverifiable, which is the failure this product exists to prevent.
Reversal condition. A request from anyone named.
Judgement. Outreach goes out under a truthful founder framing.
Why. To a utility deputy general counsel, a programme title reads like a placement; "building this" reads like a company. The truthful claim is also the stronger one, and it needs nobody's permission. A third party's name is used only with that party's agreement, and never in messages to commission staff or consumer advocates who sit opposite that party in live dockets.
Judgement. Decline the number; propose a stronger standard.
Why. Line coverage measures execution, not verification. The line that rejects a fabricated quote can be green while nothing asserts that it rejects anything. The substitute: total branch coverage on the four deterministic load-bearing modules, mutation testing to prove the tests catch defects, property-based tests for invariants, and evals rather than coverage for the model path.
What happened next, and what it corrects in the judgement above. The owner overruled it, asked for both standards at once, and later cut the 99% target on cost. The judgement was right about what line coverage measures and wrong in one respect worth recording: at the time it was made, none of the substitute existed and no coverage had been measured either, so declining the number was also declining to answer. A coverage tool went in during the override and the figure became 92%. It is reported as a measurement, not adopted as a standard.
Judgement. Do not send 45 near-identical emails in one evening.
Why. The targets cluster into about twelve organisations — four at Duke, five at the OUCC, three at Georgia PSC. Regulatory affairs is a small, connected world and colleagues forward things. Three people at one firm receiving the same message reads as a blast, and that impression is hard to undo with the exact audience whose trust the product needs.
Thirteen defects. Nine were found by reading or attacking the code rather than by a test in the suite going red, and three were false statements in documents rather than bugs in code. Two conclusions follow, and both are worth defending rather than hiding.
First, the test suite is not the safety net it looks like. Fifty-six passing tests did not catch an unscoped tenant read, because no test asked the question. That is the argument for mutation testing over a coverage percentage, and it is why finding #1 is described here rather than quietly fixed. Findings 12 and 13 say the same thing at four hundred tests instead of fifty-six: a green suite over a module whose every branch runs still missed two defects, because both live in text a person does not think to write. Size did not fix it. A fuzzer did.
Second, a document is a claim, and this project holds claims to a standard. The three false statements above were all written in good faith by people describing what they intended to build. That is exactly how a model produces a confident wrong answer, and it is why the product's own answer — verify, and withhold what does not verify — is applied to its own prose.