A parsed document hierarchy, measured and not wired in

A negative result. Every figure on this page is re-derived from the committed bytes of data/real by one command, and a test fails if this page and that command disagree.

.venv/bin/python scripts/measure_structure_signal.py

Why this was measured at all

app/diff/engine.py::_alignment_confidence decides how much to trust that two passages are the same passage. Its only structural evidence is the integer at the head of each one, and its own docstring has always said that is a proxy. This page is what happened when the proxy was measured rather than argued about.

Among the labels _section_label hands back are 2024, 2025, 0.0000, 207, 41 and 480. Some are years, one is a table cell and the rest are line numbers off a deposition page. Where two such numbers disagree, the engine caps alignment confidence at RESTRUCTURE_CONFIDENCE_CEILING and the change escalates. That is the right behaviour when the numbers are section numbers. It is a queue full of alarms when they are not. That problem is real and this page does not solve it. The last section says what would.

Part A — what the parser can read

27 of 102 real filings yield a hierarchy. The rest are refused, each with a reason naming the evidence. All three synthetic proceeding versions parse.

A refusal rate of three in four is not a defect being tuned away. Most of this corpus genuinely has no hierarchy a machine can see: a two-page errata cover letter, a semi-annual load report that is one table, a motion whose only structure is paragraph breaks. Inventing a tree for those would be the confident wrong answer ADR-003 forbids. An unparseable document yields no tree, never a flat one pretending to be a tree — and a DocumentTree with zero roots raises rather than constructs, so there is no fifth outcome for a future refactor to slip into.

The document the nesting rules were learned on

in-45911-final-order.txt is the best real nesting case in the corpus: I. → A. → 1. → 1.1, four levels, with the label and its title on separate lines throughout because the text came out of a PDF. It is also 382,479 characters that _segment cuts into 238 passages, the longest of them 17,433 lines. A parser that treated one passage as one node could not see this document at all, which is why app/diff/structure.py parses lines inside passages and carries absolute offsets.

What each rule bought, measured by deleting it

Each figure below comes from taking that one rule out of app/diff/structure.py's own source, importing the result as a second module, and re-parsing all 102 filings. Not from a hand-written copy of the rule with the condition flipped, and not from memory. Four of this page's earlier figures came from memory and none of the four reproduced.

The gutter detector, and the one thing that makes it believable

A document is called line-numbered when a set fraction of its non-blank lines open with a small integer followed by text on the same line, and somewhere in it that many such lines run consecutively both in position and in value. The two constants are GUTTER_LINE_FRACTION and GUTTER_MIN_ADJACENT_RUN.

With exactly those two values, 19 filings are flagged, and every one of them is direct or rebuttal testimony. That is the check that makes the thresholds credible — not the numbers themselves, but the fact that the set they select has a name. A test pins it: if a tuning change ever flags an order, a tariff or a settlement, the constants have stopped meaning what this page says they mean.

The detector errs where erring costs least. There are 33 filings with "testimony" in the name and it misses 14 of them, whose gutters did not survive PDF extraction as leading integers. It has never flagged a non-testimony document on this corpus.

Two details in the rule, one bought with a document and one that has never decided anything:

• The number must be followed by text on the same line. in-45911-final-order.txt contains an extracted table rendered as a bare column of ascending integers, nothing else on the line. Without the trailing requirement that column reads as a 33-line gutter run and the document is refused; with it the file's hit fraction falls from 0.35 to 0.03 and its four-level hierarchy survives.
• The run must be adjacent, not merely consecutive in value — and on this corpus that condition never decides anything. An earlier version of this page said it was bought with a document, oh-2025-1458-merit-brief-puco.txt, and gave a run length nothing in this repository produces. What that file really measures: a value-run of 21 against an adjacent run of 4, at a hit fraction of 0.0154 — an order of magnitude under the threshold, so the fraction had already refused it and adjacency never came into it. Across the corpus, 0 filings clear the fraction and are saved by the run, and 2 clear the run and are saved by the fraction. The smallest adjacent run among the flagged is 21, against a constant of ten. The condition is kept because it can only ever flag fewer documents and flagging an order would be the expensive mistake, but it is a lock that has never turned and the code says so where the constant is defined.

One line, one heading

I is the ninth letter and it is also the numeral one, and no regex settles which. Both readings could be accepted at once, and in in-46258-final-order both were: the offset 102,139 entered the tree twice, at depth 0 as ('I',) and at depth 2 as ('H','2','I'), and the deeper of the two came out with char_end == char_start — a node of zero width that path_at can never return. The document reported 113 headings and offered one fewer, and the higher number is what an earlier version of this page published.

The rule now is that the reading with the longer sibling run wins: at that offset the alpha run is 9 and the roman run 3, so the letter wins, which is right — the line is titled COSS/Rate Design and its neighbours are H. and J. Acceptance is then re-run without the losing reading rather than the loser being deleted after the fact, because that I was also the value-1 that opened the roman sequence: delete it and roman is left holding II and III, a run of two, and the scheme goes with them — 110 headings and no roman at all. Re-running lets roman open instead at the I. at offset 130,193. The document reads 98 headings today.

There is no gold set here. What can be checked is that no node is unreachable and no offset is claimed twice, and a test checks both across all 27 parsed filings.

Part B — what wiring the signal in would have changed

These 8 are the pairs in data/real that really are two versions of one document. scripts/ingest_dockets.py sets out in its own comments why two other look-alikes are not pairs, and it is right. For each pair the shipped confidence is compared against the confidence the proposed rule would have returned. The proposed rule now lives in the measuring script under the name _withdrawn_confidence, which is the only honest place for it: a branch kept alive in app/ so a measurement can reach it is a branch that gets re-wired by accident.

Why that is a reason not to ship it

All the movement is in one document pair, and one pair is not a corpus. The other seven produced nothing at all: six because the parser refused both sides and there was nothing to say, and the seventh — Missouri's stipulation, the one pair where both sides yield a tree — because the hierarchy left all 3 of its changes exactly where the proxy had them.

The alarms are real. This was the wrong instrument for them.

112 escalations produced by a shifted gutter number are 112 alarms an analyst opened and dismissed. 2 of them are word-for-word identical passages three words long, where the gutter number is a large fraction of the characters:

ut-24-035-04-laconte  BEFORE '249          Account.'   AFTER '250          Account.'
ut-24-035-04-laconte  BEFORE '292         made:'       AFTER '293         made:'

That is worth fixing. The fix is not a parsed hierarchy standing behind the proxy and overruling it on the strength of its own failures. It is _section_label learning that a page with a line-number gutter has no section numbers on it — a change to the proxy, on the proxy's own terms, with none of this machinery in the confidence path. The gutter detector in app/diff/structure.py is the part of this work that would be reused, and it is measured above. That change is not made here: it moves 112 real confidences and it needs its own decision record, its own tests and its own review, not a paragraph at the end of a page about something else.

Part C — the renumbering case

I did not find a real renumbering in the eight version pairs. The search parsed both sides of every pair and looked for a heading whose title survives while its path changes. It found nothing. That is not a claim that none exists — the corpus was not read by eye, and a title match cannot see a heading that also had its wording edited. It is a claim about what this search found, which is nothing.

So the demonstration below is SYNTHETIC, built from data/v2_revised_proposed_rule.txt and data/v3_final_order.txt, and it is labelled synthetic here, in the script's output, and in the commit that added it.

SYNTHETIC before: 'SECTION 6. COLLATERAL AND FINANCIAL ASSURANCE'
SYNTHETIC after : '5.4 Collateral and Financial Assurance.'
SYNTHETIC v2 path at the heading: ('6',)
SYNTHETIC v3 path at the heading: ('5', '5.4')
SYNTHETIC withdrawn verdict: different-node
SYNTHETIC confidence, shipped: 0.5000
SYNTHETIC confidence, withdrawn rule: 0.5000

Both the shipped proxy and the withdrawn rule cap this one. That is the point, and it is also why the tree was never load-bearing here: the proxy was already doing it. A tree that rescued this change — that decided SECTION 6 and 5.4 were one node because their titles match — would be a tree that had learned to agree with the mistake ADR-004 exists to prevent.

Limits

• Every constant was tuned on this corpus, in one sitting. GUTTER_LINE_FRACTION, GUTTER_MIN_ADJACENT_RUN, MIN_SIBLING_RUN, MIN_HEADINGS. The evidence for them is one plausibility check — the documents they select are all testimony — and that is a check, not a validation. They will be wrong on a jurisdiction that is not in data/real.
• There is no gold set for document structure. Nobody has labelled which lines of these filings are headings. Part A therefore reports how many documents the parser will read and why it refuses the rest. It cannot report accuracy, and this page does not.
• The heading counts are pinned by a test, and that test is the only thing standing behind them. tests/test_structure.py holds a table of every parsed filing with its heading count and scheme order, so a change to any parsing rule fails a test and has to restate what it changed. It was added after a reviewer deleted three rules that shape the tree and watched the whole suite stay green while five documents changed shape and two changed the depth of every heading they have. The parse rate is unmoved by each of those deletions, so no count of documents could ever have seen it.
• Where a document parses, the parser silently discards candidates it judges out of sequence. A tree can therefore be partial: it may hold nine of a document's twelve real headings and say nothing about the three it dropped. It refuses whole documents loudly and drops individual headings quietly, and the second half of that is a weaker guarantee than the first.
• Depth comes from the order schemes first appear, which is not always nesting order. In in-45911-final-order the first accepted heading is an A. that appears before the first I., so the parser puts alpha above roman where the document means the reverse. The paths are still consistent within a document, but the depths are wrong there and I do not have a rule that fixes it without guessing.
• Front matter has no path, and that is deliberate. 3,698 of the 63,758 passages in the parsed filings sit before the first heading — the caption, the service list, the notice block. path_at returns nothing for them. Comparing two nothings with == would call every caption the same node as every other caption, which is the gutter mistake in a smaller coat.
• The backward path of the proxy defect is not fixed either. app/ingestion/ingest.py::_section_of uses the same leading-number pattern and writes the result to Passage.section, which the changes screen renders as Sec 193 for a testimony line number. Correcting that re-labels stored passages across every ingested filing, which is a migration under best-practices §27, not a line change.
• I do not know whether the parse rate would rise or fall on a corpus of federal filings, tariff sheets, or anything from a jurisdiction outside the eight here. Nothing on this page is evidence about that.