What the diff path costs today, where it breaks, and the design that follows from those numbers. Nothing in section 4 is built.
tests/test_diff_scale.py::test_the_pipeline_peak_is_traced_while_it_holds_the_document now watches the boundary rather than the number.One million words of filing text is about 6.5 million characters, so a version pair of million-word documents is about 13 million characters. That is not an extrapolation here: it is a row in the tables below. The 6,400-passage point is 13.16 million characters. app/diff/engine.py::diff costs 2.599 seconds and 14.26 MB of peak Python heap on it. The call a worker actually makes, app/pipeline.py::ingest_and_diff, costs 2.930 seconds and 38.51 MB — it holds the document it was handed, the passages it segments out of it, the previous version's rows, the diff, the change rows and an audit row per change. Both pairs are one row of one table, measured back to back. The second pair is what the arithmetic below multiplies, and using the first pair was the error a reviewer found in the previous version of this page.
Ten thousand of those:
So the answer to "ten thousand concurrent" is that you never run ten thousand concurrent.
The gap between those two lines is the whole design. A bounded pool of eighteen workers holds 693 MB, which fits on a laptop, and clears the backlog in 27 minutes. An unbounded one asks for 376 GB and is killed by the operating system halfway through, leaving a half-written transaction and no explanation. Nothing in this repository bounds it. There is no pool, no queue and no admission check; the size of the corpus is what keeps it honest, and the size of the corpus is not an argument.
The genuinely binding constraint is not the ten thousand. It is one bad document. Section 2's second table shows that a run of repeated passages is quadratic. Sixteen thousand repeated lines take 15.1 seconds. A million-word filing that is mostly a rate table — say 60-character rows, so about 108,000 of them — extrapolates on that curve to roughly twelve minutes for a single pair: (108,000 / 16,000)² × 15.1 seconds. A filing of 10-character [Reserved] lines, 650,000 of them, comes to nearly seven hours the same way. Those two figures are extrapolations from the measured curve, not measurements, and they are the reason section 4's admission check refuses by name rather than trying.
Two honest qualifications, both of which matter more than the arithmetic. Every multi-worker figure above is single-worker measurement multiplied by ten thousand: nothing on this page was measured under concurrency at all, so it ignores allocator contention, the GIL and a shared database. And the pipeline figure was measured on a pair where one passage in twenty changed; record_event writes an audit row per change, so a wholesale redline pays a term this arithmetic does not carry — see section 3.
Reproduce with scripts/bench_diff.py:
.venv/bin/python scripts/bench_diff.pyThe corpus comes from an LCG written out in that file rather than from random.Random, so two runs on two machines with two Pythons compare line for line. The passages carry real cumulative offsets, because diff() puts those offsets into every citation and a benchmark that fed it zeros would exercise a shape the product never sees.
tracemalloc's: the Python heap, not the interpreter, not SQLite's page cache, not a C extension. And timing and memory are two separate passes over the same input, because tracing inflates this workload's wall clock by seven to eight times — app/text/normalize.py::normalized_projection allocates per character and tracemalloc charges every allocation. bench_diff.time_only raises rather than trusting anyone to remember that.Every passage differs from every other, 120 words each; one passage in twenty is amended on the after side.
| passages | Mchars, both sides | seconds | µs / char | peak MB | changes |
|---|---|---|---|---|---|
| 400 | 0.82 | 0.163 | 0.198 | 0.9 | 20 |
| 800 | 1.64 | 0.342 | 0.208 | 1.8 | 40 |
| 1,600 | 3.29 | 0.679 | 0.207 | 3.6 | 80 |
| 3,200 | 6.58 | 1.349 | 0.205 | 7.1 | 160 |
| 6,400 | 13.16 | 2.856 | 0.217 | 14.2 | 320 |
| 12,800 | 26.31 | 5.942 | 0.226 | 28.4 | 640 |
| 25,600 | 52.61 | 13.278 | 0.252 | 56.9 | 1,280 |
Linear across sixty-four times the input, with the caveat in the box above about the last two rows. Peak memory is linear with no caveat at all: it works out at 1.13 to 1.20 bytes of peak Python heap per source character at every point, including the largest. Read that constant with the subsection below beside it: it is what diff() allocates on top of passage lists that already exist, which is not what a worker holds.
100 unique passages, then a run of N identical [Reserved] lines with one of them amended in the middle. This is not a contrived shape: a tariff sheet with two hundred reserved lines, or a rate table whose rows repeat, is an ordinary filing.
| repeated lines | Mchars, both sides | seconds | changes reported |
|---|---|---|---|
| 500 | 0.22 | 0.055 | 2 |
| 1,000 | 0.23 | 0.099 | 2 |
| 2,000 | 0.25 | 0.272 | 2 |
| 4,000 | 0.29 | 0.976 | 2 |
| 8,000 | 0.37 | 3.791 | 2 |
| 16,000 | 0.53 | 15.129 | 2 |
Doubling the run approaches quadrupling the time: 1.8x, 2.8x, 3.6x, 3.9x, 4.0x across the five doublings. The early ratios are below four because the linear normalization cost still dominates at 500 lines; by 8,000 the quadratic term owns the call. Note the document sizes: 525,059 characters here cost more than the 52.6 million characters of ordinary prose in the table above, which run in 13.3 seconds. Size in bytes is not the variable; repetition is. Memory was not measured for this family — the cost is in the matcher's comparisons rather than in allocation — and the table says so rather than carrying a figure from the other one.
Kentucky PSC 2025-00113, Lane Kollen's direct testimony as filed and as corrected by the filer, segmented the way app/ingestion/ingest.py::_segment segments it.
| passages | Mchars, both sides | seconds | µs / char | peak MB | changes |
|---|---|---|---|---|---|
| 4,285 | 2.03 | 0.403 | 0.199 | 2.3 | 144 |
1.17 bytes of peak heap per source character, in line with the synthetic constant on a document nobody wrote for this benchmark.
docs/tdd.html quotes for the same pair, and both are right. That figure is the whole pipeline: ingestion, segmentation, the diff, and the passage, change and audit rows written to SQLite. The 0.403 seconds here is diff() alone, in memory, with no session and no writes. Anyone comparing the two is comparing a function with a transaction — and the subsection below now measures the transaction, so the comparison no longer has to be guessed at.diff() allocates on top of passage lists that already exist — tracing starts after
the caller has built them, so the documents are outside the traced region by construction. That is the right
figure for how the call grows with its input and the wrong figure for sizing a worker, and an earlier version
of this page used it for the second. It understated the real call by a factor of 2.6 at the smallest size
measured and 2.7 at the largest, in the direction that admits a pair a worker then cannot hold.bench_diff.footprint_measurement traces three regions on the same input, each wider than the
last. The third is app/pipeline.py::ingest_and_diff against a scratch SQLite file: the new
version's source string joined inside the traced region, then segmentation, the previous version's rows read
back out of SQLite, the diff, the change rows and an audit row per change. That is the call a worker
makes.
| passages | Mchars, both sides | diff() alone — MB and bytes/char | the inputs and diff() | the whole pipeline call | diff() seconds | pipeline seconds | changes |
|---|---|---|---|---|---|---|---|
| 400 | 0.82 | 0.95 MB — 1.21 | 1.50 MB — 1.91 | 2.46 MB — 3.14 | 0.153 | 0.172 | 20 |
| 1,600 | 3.29 | 3.61 MB — 1.15 | 5.83 MB — 1.86 | 9.69 MB — 3.09 | 0.616 | 0.694 | 80 |
| 3,200 | 6.58 | 7.16 MB — 1.14 | 11.59 MB — 1.85 | 19.42 MB — 3.10 | 1.241 | 1.395 | 160 |
| 6,400 | 13.16 | 14.26 MB — 1.14 | 23.13 MB — 1.84 | 38.51 MB — 3.07 | 2.599 | 2.930 | 320 |
The pipeline constant is flat at 3.07 to 3.14 bytes of peak Python heap per source
character across sixteen times the input, which is what makes an admission check arithmetic rather
than a guess. Section 4 rounds it up to 3.5 and says
why. tests/test_diff_scale.py::test_the_page_never_admits_a_pair_bigger_than_a_worker_can_hold
fails if that published number ever falls below what the pipeline measures, and
::test_the_pipeline_peak_is_traced_while_it_holds_the_document fails if the document is ever
built outside the region again. That second guard is not an inequality on the bytes, because none would work:
moving the join out costs about half a byte per source character and the region has more slack than that, so
any threshold that separated the two would be a number tuned to one run. It watches where the allocation
happens instead.
Both second columns are measured with tracing off, back to back, in a pass of their own away from the
peaks. They are in the same row so that the two can be compared with each other: the page used to set 3.209
seconds for the pipeline against 2.856 for diff() and take the difference for the cost of the
transaction, and those two figures came from two runs at two load averages on a laptop this page says was
never idle. A ratio between two runs is not a ratio. On this row the pipeline call costs 1.13 times
diff() alone at the largest size. docs/tdd.html quotes
0.78 seconds for the pipeline on the real Kentucky pair, which is the same kind of measurement at one size.
Three things these seconds do not tell you. The first is small and is the price of the correction above:
joining the source document happens inside the timed region, because it has to happen inside the traced one,
and a worker is handed its text rather than joining it. The other two would push the figure up. The fixture amends one passage in twenty, so the largest pair here
carries 320 changes; record_event writes an audit row and flushes for
every one of them, so a pair where most passages changed pays a term this table cannot see. And
tracemalloc counts the Python heap, so SQLite's own allocations are outside every figure on this
page. Both are on the list in section 6.
The reviewer's premise — that SequenceMatcher is the constraint — is wrong on this codebase's shapes, and it is the most useful thing the measurement turned up. Split at 1,600 passages, 3,287,820 characters both sides:
| stage | seconds | share of the call |
|---|---|---|
normalize() over both passage lists | 0.550 | 87% |
the list-level SequenceMatcher | 0.0096 | 1.5% |
_alignment_confidence over the 80 modified pairs | 0.070 | 11% |
whole diff() | 0.630 | 100% |
The cost is app/text/normalize.py::normalized_projection: a per-character Python loop that folds quotes and dashes, deletes soft hyphens, resolves hyphenated line breaks and carries a raw offset for every output character. It is O(n) and it is correct; it is simply written in Python. Anybody who reads "SequenceMatcher is quadratic", opens app/diff/engine.py and starts tuning the matcher will spend a day and find nothing.
The third row used to be an estimate and was wrong by 37%. The page said this fixture had 80 modified pairs of about 750 characters each; they average 1,027. The stage is now timed over the pairs the diff produced rather than inferred from a per-character rate and a length quoted from memory.
Read those shares to the nearest few per cent and no closer. Three runs of the same measurement, back to back at the same commit and the same load, gave normalization 87 to 88 per cent, the matcher 1.5 per cent every time, and _alignment_confidence 11 per cent every time. The stages and the whole call are timed in separate passes, so they do not have to add up: across those three runs the unattributed leftover ran from -3.8 to +1.0 milliseconds on a call of about 0.63 seconds, and it changes sign. A leftover that changes sign is timing noise, not a fourth stage. The earlier version of this table, measured on a busier machine, put normalization at 84 per cent and the leftover at +22 milliseconds; the conclusion the section rests on is the same one either way, and it gets stronger as the machine gets quieter.
app/diff/engine.py::diff passes autojunk=False. difflib's default treats any element appearing in more than one per cent of a sequence longer than 200 as junk and never matches it. On a filing, the element appearing three hundred times is the [Reserved] line, and junking it collapses the alignment inside the run. Measured on 100 unique passages plus a run of 300 identical ones with a single amendment:
| setting | changes reported | pairs whose before and after read alike | distinct before-passages named | matcher seconds |
|---|---|---|---|---|
autojunk=False, shipped | 2 | 0 | 1 | 0.0048 |
autojunk=True, difflib's default | 150 | 149 | 150 | 0.0001 |
Tens of times faster and wrong — the table gives both matcher timings, and the ratio between two measurements this small is not worth a decimal place. One amendment becomes 150, of which 149 pair a passage with a passage carrying exactly the same text: a change screen full of amendments nobody made. Nothing downstream can catch it, because the citation on a fabricated change verifies — the words really are in the document. The diff is the last place the truth is known.
So the quadratic curve above is the price of that setting, and it is worth paying. tests/test_diff_scale.py fails if it stops being paid.
app/diff/engine.py::_alignment_confidence runs a character-level SequenceMatcher for every modified pair. It appears in neither curve because neither shape produces many long modified pairs — the repeated-run fixture produces none at all. Measured per call:
| passage characters | ms per call | µs / char |
|---|---|---|
| 190 | 0.345 | 1.81 |
| 540 | 0.514 | 0.95 |
| 1,548 | 1.886 | 1.22 |
| 4,590 | 4.668 | 1.02 |
About one microsecond per character, against 0.21 for the whole document path — so this term costs roughly five times as much per character as everything else, and it is charged only on modified pairs. A version pair where most passages are modified and long is dominated by this function rather than by either curve above. No shape in the current corpus does that, so it has never mattered; a redline of a whole tariff would.
app/pipeline.py::ingest_and_diff holds both versions' passage lists in one session, builds two section maps over them, and calls record_event once per change — a flush and an audit row each. That path, not diff(), is what a production worker runs. The subsection in section 2 now measures it: on the 6,400-passage pair it costs 2.930 seconds against 2.599 for diff() alone, timed back to back in one pass, and 38.51 MB of Python heap against 14.26. What is still not measured is the shape that would make the transaction the ceiling: a pair where most passages changed. The fixture amends one passage in twenty, so the audit write per change is a small term in every figure above. A wholesale redline would make it the large one, and nothing here says how large. Section 6 keeps that on the list.
The unit is (company_id, proceeding_id, previous_version_id, version_id) — the argument tuple of ingest_and_diff. Not a document, and not a passage. The reason is in the code rather than in taste: app/pipeline.py::change_id derives a change's identity as CHG-{from}-{to}-{ordinal:03d}, where ordinal is the position in the whole-document diff output.
You cannot split a version pair across two workers without changing that derivation, because the second worker cannot know its ordinal offset until the first has finished. And changing the derivation is not a refactor, it is a corpus migration: rows already stored keep the old ids, a re-run mints new ones, and the row count doubles. That is the failure change_id's own docstring describes, and best-practices §27, a derived corpus migrates all at once.
The parallelism available here is therefore across pairs, and there is plenty of it: 19 dockets and 105 versions today, and a real customer has thousands of proceedings. A design that split one pair to make one filing faster would be buying latency on one screen with the identity of every change in the database.
ingest_and_diff peaks at 3.07 to 3.14 bytes of Python heap per source character at every size measured, and that is the whole call — the document it was handed included — rather than the diff inside it. The constant is the point of measuring: a worker's footprint is predictable from the two versions' byte counts before it starts.
So an admission check would be arithmetic, not a guess. A pool of N workers under a total cap of M bytes admits a pair when 3.5 * (len(before) + len(after)) < M / N — 3.5 rather than 3.14, because the constant is rounded against the product and has to cover the spread between runs rather than the best of them. That is eleven per cent of headroom, not thirty. The thirty was arithmetic against 2.69, and 2.69 was measured on a region that did not hold the source document; when the region was widened to hold it the round-up stopped being generous and became close. It still covers the measurement, which is the only thing the guard below asserts, but a pool built on it should be sized knowing the margin is one part in nine. The same measurement taken from inside the test suite, on the same laptop, reads 3.12. A pair too large for any worker is refused by name: the docket, the version and the size that would not fit. Not started and killed. A worker killed by the operating system leaves a half-written transaction and no sentence anybody can read.
That bound covers ordinary documents. It does not cover the quadratic case, where the cost is not predictable from byte count at all — 530,000 characters of repeated rows cost more than fifty-two million characters of prose. An admission check on bytes alone would wave that through. A second check would have to count distinct normalized passages, which is one pass over the document and is the only cheap predictor of the quadratic term I know of. Neither check is implemented, and the second one is a guess I have not tested.
Three tests hold that arithmetic to the code. tests/test_diff_scale.py::test_diff_holds_close_to_one_copy_of_the_document_in_memory fails if something inside diff() starts keeping a per-character structure alive. ::test_the_page_never_admits_a_pair_bigger_than_a_worker_can_hold fails if the 3.5 above ever drops below what the pipeline measures; it exists because the first version of this page published 1.2, derived from a peak that excluded the documents themselves, and nothing caught it for a day. ::test_the_pipeline_peak_is_traced_while_it_holds_the_document fails if the document is built outside the traced region; it exists because the fix for the first fault reproduced it one region wider, and the second time nothing caught it either. Both of those errors run the same way round — they understate what the machine has to find — and this is the one arithmetic on the page where erring low gets a worker killed.
Between the fetch and ingest_and_diff. Not inside diff.
app/jobs/runner.py is 1,242 lines of real scheduler and this design has to sit beside it rather than quietly duplicate it. What it already gives: JobRunner.run_once sweeps every registered Job across every company on the roster; exclusive(lock_path_for(...)) takes a file lock derived from the database URL, so two processes cannot sweep at once; JobRunner.failing counts consecutive failures per (job, company); JobLoop runs the sweep on a thread; and every sweep writes audit rows.
What it is not, and this is the distinction worth being able to state out loud: the JobRunner runs cadences, not work items. No unit of work with an identity, no payload, no retry of a particular item, no ordering between items, and no producer that can outrun it — a sweep either happens or does not, and the next one starts from the current state of the database rather than from a backlog. A queue is precisely the thing that gives a unit of work an identity, so it can be retried, deferred, refused or counted.
The producer side is already honest about itself. JOB_SOURCES is a refused job name: app/jobs/runner.py::_SOURCES_NOT_BUILT says there is no crawler and no poller and that ScheduledRun is a table of cadences with nothing behind it. app/sources/fetch.py::fetch_source does exist and is heavily tested — it retrieves one registered filing, hashes it, and stores it only if it changed — but nothing in the running product calls it; its only callers are in tests/test_fetch.py. The producer today is a function with no caller. So backpressure here is a design constraint on a component nobody has written, and saying anything more confident would be describing a system that does not exist.
The house answer is the only one available, and it follows from "absence is denial": when the queue is full, or a pair fails admission, the producer is refused, by name, with the reason. Never dropped. Never silently deferred to a later sweep nobody is watching.
The reason is a product fact rather than an engineering preference. A pair that was not admitted has no changes, and a change screen with no changes looks exactly like a filing that did not change. That is the worst failure available to this product: a regulatory analyst reads "no changes", closes the tab, and the amendment they are accountable for is sitting in the document. So an unadmitted pair has to be visible as an unadmitted pair — on the proceeding, in the audit trail, and on whatever screen the analyst is reading. Same rule that made JOB_SOURCES a refusal instead of a job that does nothing; same rule as best-practices §26, a fallback must announce itself.
Anchors, not indices — and the measurements in the next section say the obvious version of this does not answer the measured problem. It belongs there rather than here, because the interesting part of partitioning is not the mechanism, it is what it costs in correctness.
normalize() is 87% of the diff call and that a version takes part in many diffs, which
is a caching problem stating itself out loud, and this page quoted that number as evidence about
where the cost lives without asking why it is paid more than once.The proposal. Hash each normalised passage at ingest. At diff time compare the two sequences of hashes, take the hashes that appear exactly once on both sides as anchors, and align only the gaps between them. Equal hash means unchanged; unequal hash means nothing at all.
Measured on the Kentucky pair — 4,285 against 4,287 passages, 1,024,409 against 1,024,536 characters, the errata refiling whose two versions differ by 127 characters. Every row produced the same 144 changes as the baseline.
| Approach | Time | Against baseline |
|---|---|---|
Baseline, diff() as it ships | 0.404 s | — |
| Hashing at diff time, prefix and suffix trimmed | 0.610 s | 0.65x — slower |
| Hashes from ingest, prefix and suffix trimmed | 0.307 s | 1.32x |
| Hashes from ingest, anchored on unique hashes | 0.171 s | 2.36x |
The losing row is the important one. Computing the hashes inside the diff costs 0.297 s, which is most of the 0.404 s it was meant to save, because hashing a passage means normalising it and normalising is the 87%. The pre-pass is only a saving if the hash is written once at ingest and read at diff time. As a diff-time optimisation it is a loss, and a plausible-sounding one.
Prefix and suffix trimming alone is weak, and the reason generalises. It removed 50.85% of the input — 1,903 identical passages at the head, 276 at the tail — and returned 1.32x. The 144 changes are scattered through the document rather than gathered at one end, so trimming the ends leaves 2,106 passages in the middle. Anchoring works between every pair of anchors instead of only at the edges, which is why it doubles the gain: 3,414 usable anchors, 286 gaps, each gap small enough that the superlinear term never gets going.
Repeated boilerplate cannot anchor, and that is the safety property rather than a
limitation. 515 of the 4,285 passages on the before side — 12.0% — share a hash
with another passage in the same document. A rule that matched on hash equality alone would pair the
wrong occurrences. Requiring a hash to be unique on both sides excludes every one of them by
construction. This is the same trap autojunk fell into by treating repeated elements as
noise, which section 7 measures at 150 reported changes where 2 occurred.
What the synthetic case says, and why it is not the headline. Over generated corpora of 1,000 to 16,000 passages with 0.5% edited, the same anchoring returns 30x to 42x with identical change counts. That number is not quoted above because the generator makes almost every passage unique, which is the best case anchoring can ever have. The real filing returns 2.36x. Where the two disagree, the filing is the evidence and the generator is a ceiling.
app/diff/engine.py; there is no hash column on passages and adding one
is a schema change plus a derived-corpus rebuild, which is the migration rule in playbook §27.
Unmeasured: the memory profile of holding two hash lists beside two passage lists; the wholesale
redline where no hash matches, which pays the comparison and saves nothing and belongs in the
admission check rather than here; and whether anchoring changes what
_alignment_confidence reports, since aligning inside a gap gives the matcher a different
neighbourhood from aligning across the whole document. The change counts matched on every case tried;
the confidence values were not compared.Partitioning changes what the matcher can see: a passage in chunk three cannot align to a passage in chunk one. That much is obvious. What it costs is not, so it was measured rather than asserted. scripts/bench_diff.py::partition_measurement runs both paths over the same input and compares the change sets by identity — (change_type, before offset, after offset) — never by count, because two different change sets of the same size are still two different answers.
tests/test_diff_scale.py::test_the_partition_measurement_calls_two_different_answers_different runs the same code over the smallest fixture with that property.Index partitioning cuts every k passages and aligns the chunks by position. Anchor partitioning cuts only at passages whose normalized text occurs exactly once on each side, with the singletons in the same order, and takes every s-th such passage.
| case | whole document | index k=4 | k=8 | k=16 | k=32 | anchor s=5 | s=50 | s=200 | s=1000 |
|---|---|---|---|---|---|---|---|---|---|
| shipped corpus v1→v2, 44 passages | 8 changes, 0.005s | 8 | 8 | 8 | 8 | 8 (7 cuts, 0.008s) | — | — | — |
| shipped corpus v2→v3, 44 passages | 19 changes, 0.007s | 19 | 19 | 19 | 19 | 19 (5 cuts, 0.010s) | — | — | — |
| real KY pair, 4,285 passages | 144 changes, 0.431s | 777 | 612 | 377 | 260 | 144 (682 cuts, 0.787s) | 144 (68 cuts, 0.796s) | 144 (17 cuts, 0.792s) | 144 (3 cuts, 0.794s) |
| synthetic 3,200 unique passages | 160 changes, 1.401s | 160 | 160 | 160 | 160 | 160 (608 cuts, 2.525s) | 160 (60 cuts, 2.470s) | 160 (15 cuts, 2.485s) | 160 (3 cuts, 2.499s) |
| boilerplate run of 8,000, 8,100 passages | 2 changes, 3.833s | 2 | 2 | 10 | 10 | 2 (20 cuts, 3.999s) | 2 (2 cuts, 4.060s) | 2 (0 cuts, 4.052s) | 2 (0 cuts, 4.035s) |
| 40 passages, one 8-passage section moved to the end | 16 changes, 0.005s | 27 | 27 | 27 | 24 | 38 (4 cuts, 0.010s) | — | — | — |
data/v1..v3 is in place, and the only length change is at the tail, so no chunk boundary ever has to move. Presenting those four "same" cells as evidence that partitioning is safe would be presenting the easiest input in the repository as a general result. A reviewer who noticed that themselves would be right to stop trusting the rest of this page.Anchored partitioning made nothing faster. The real pair: 0.787 seconds partitioned against 0.431 whole. The synthetic pair: 2.53 seconds against 1.40. About 1.8 times slower in both cases, and the reason is section 3 — finding anchors normalizes the whole document once, and then every sub-diff normalizes its slice again, so the partitioner pays the linear term twice, and the linear term is around 87% of the call. Any partitioner worth building must take pre-normalized passages, which means diff() would have to accept them. Today it cannot: it normalizes internally, which is exactly what makes it safe to call from anywhere.
Partitioning does not touch the quadratic case at all, and this is the one to lead with. The shape that goes quadratic is a run of repeated passages, and a repeated passage is by definition not unique, so it yields no anchors. On the 8,100-passage boilerplate document the anchor pass found 20 cuts at stride 5 — all of them in the 100-passage unique prefix — 2 cuts at stride 50, and none at all at strides 200 and 1,000. The cost was unchanged: 4.00 seconds against 3.83. The obvious design does not answer the measured problem. That is the single strongest argument on this page for measuring before building.
Stated precisely, because the loose version of this claim is what a reviewer will probe.
diff() is a pure function and stays one under partitioning. The same two documents cut at the same points give the same changes every time, on any machine. Nothing about partitioning introduces a clock, a hash seed or a thread race.
What partitioning weakens is something else, and it is worth naming exactly: the answer stops being a function of the two documents alone and becomes a function of the two documents and the cut points. Two runs that choose different anchors — because the stride changed, or because the anchor rule was improved — give different change sets, and the change ids derived from ordinals move with them. So under partitioning the cut points become part of the input. They would have to be stored beside the changes, or the result is no longer reproducible from the corpus, and "re-run the loader and get the same database" stops being true. Today the corpus alone reproduces the answer. That is a real weakening and it should be paid for only by a measured speed-up, which section 5 has just shown does not currently exist.
What would detect a regression: running both paths over the corpus and comparing change sets by identity. That is exactly what partition_measurement does, and it is about twenty lines. The comparison itself is in the suite — one test drives bench_diff.index_partition_rows over the fixture where the counts agree and the answers do not — but the sweep across all six cases is not, because there is no partitioner to guard yet. It exists to make the claims on this page checkable, and it becomes the guard on the day one is built.
app/pipeline.py::ingest_and_diff is now measured, and only on one shape: a pair where one passage in twenty changed. record_event writes an audit row and flushes per change, so a wholesale redline pays a term none of these figures contains, and I do not know how large it is. Nor does tracemalloc see SQLite's own memory, so the pipeline peaks are a floor for a real worker's resident set rather than a bound on it.tracemalloc sees the Python heap. It does not see the interpreter, SQLite's page cache or any C extension, so a real worker's resident set is larger than these peaks by an amount this page does not know. Where RSS is quoted anywhere, note that resource.getrusage reports ru_maxrss in bytes on macOS and kilobytes on Linux; I verified the macOS side and not the Linux one.tests/test_diff_scale.py asserts an outcome of diff() rather than difflib's internals, which is the right level, but a future CPython could move the numbers on this page without moving the guard._similarity now passes autojunk=False and scripts/remeasure_alignment.py recomputed every stored row, which is what made the change safe to make at all. 333 modified changes read, 25 moved, 2 cautions lifted, 0 added. The prose below is kept in the present tense it was written in rather than edited into the past, because it is the record of what the instrument reported before anything was done about it.app/diff/engine.py::_similarity builds SequenceMatcher(None, a, b) and takes difflib's default, which is autojunk=True. Here b is a passage's characters, so any paragraph over 200 characters is past the cutoff, and every character appearing in more than one per cent of the positions is junked. In English prose that is most of the alphabet. The ratio is then computed on what survives — punctuation, digits and rare letters — and the number that reaches the change screen is a similarity between the wrong things.
Measured on real filing text from data/real/: paragraphs changed by exactly one word, 200 per band, in four length bands because the effect depends on length (a longer paragraph raises the one-per-cent bar and junks fewer characters).
| paragraph length | n | scored ≤ 0.50 by the shipped code | worst shipped score | true similarity of that pair | mean shipped | mean true | cost of autojunk=False |
|---|---|---|---|---|---|---|---|
| 200–400 chars | 200 | 2 | 0.444 | 0.964 | 0.957 | 0.980 | 0.0001s → 0.0009s |
| 400–800 | 200 | 0 | 0.515 | 0.992 | 0.970 | 0.990 | 0.0002s → 0.0028s |
| 800–2,000 | 200 | 1 | 0.500 | 0.994 | 0.984 | 0.995 | 0.0006s → 0.0168s |
| 2,000–20,000 | 200 | 8 | 0.480 | 0.992 | 0.974 | 0.996 | 0.0262s → 1.0911s |
LOW_ALIGNMENT in app/web/views/changes.py is 0.50. 11 of the 800 real paragraphs sampled — one in 73 — cross that line after a one-word edit, and the worst of them scores 0.444 against a true similarity of 0.964. Each of those would carry a low-alignment caution on the change screen that the text does not support. No score was exactly zero in this sample.
The direction is safe, which is why this is a finding and not an incident. Junking only removes matches, so the defect can only push a score down: it over-flags and it can never under-flag. A change whose text really did move a long way cannot be rescued into a high score. So it fails toward review, as ADR-003 requires. What it costs is trust in the caution — a reviewer who keeps seeing the low-alignment flag on changes that plainly moved one word learns to ignore the flag, and then the flag is worth nothing on the day it is right.
Why no existing test can see it: every fixture in tests/test_diff.py is under 200 characters, so difflib's cutoff never fires and the ratio is exact.
Why it is not fixed here. The fix is autojunk=False in _similarity too, and it costs between 9x and 42x on the character path depending on passage length. More importantly it rewrites alignment_confidence on every stored change row, which makes it a derived-corpus migration (best-practices §27) rather than a one-line edit: until the corpus is reloaded, the screen, the stored rows and this page would each say something different. It needs its own decision and its own reload.
Where it is written down: here and in the findings table, and the ADR is still missing. This page once closed by saying the defect was recorded as open in docs/.ai/findings.html. It was not — grep that file for autojunk and it returned nothing, and the change that wrote this page had not touched it. A page pointing at a tracker that does not carry the defect is worse than one admitting the defect is untracked, because the first stops anybody looking. It is finding 17 in that table now, marked open, with no guard and the reason there is none. What is still missing is the decision itself: not fixing a live defect because the fix is a corpus migration is a choice, and a choice with alternatives and a cost belongs in docs/.ai/decisions.html rather than in the resolution column of a table.