Run 2026-08-05 at 17:15 UTC against the full gold set in data/evals/. Working tree at 7b13738, with app/evals/obligations.py and data/evals/ still uncommitted. Python 3.12.13.
--no-extractor, which records a refusal for all 51 passages. A system that asserts nothing cannot assert wrongly, so the blocking metric passed by asserting nothing. The scorer says this itself, in capitals, at the top of its own output, and prints "do not quote the exit code of this run". Correct assertions: also 0. Half the scorecard is blank because nothing has been built to fill it.| Question | Answer |
|---|---|
| Command | .venv/bin/python -m app.evals.obligations --no-extractor |
| Exit code | 0. Meaningless here, and the run note says so. |
| Without that flag | .venv/bin/python -m app.evals.obligations exits 1 and prints "Nothing to score: no --predictions file was given." It refuses to default to anything. That refusal is the correct behaviour and it is what a reviewer meets first. |
| Predictions file | None. Searched the whole tree for a JSON file holding an extractions list: none exists, tracked or untracked. |
| Extractor | None. app/interpretation/propose.py judges whether one diffed change is material; it is not asked what duty a passage creates. Nothing else in app/ does that job. |
| Gold sets read | goldset_ga_ky.json (25 passages, GA and KY) and goldset_ut_mo.json (26 passages, UT and MO). 51 passages over 16 source documents. |
| Scorer's own tests | .venv/bin/python -m pytest tests/test_obligation_evals.py -q — 62 passed in 0.41s. |
Wired into make eval | No. That target still prints the five spine metrics. This is a second scorecard and runs on its own. |
| Outcome | Count | What it cost |
|---|---|---|
| Wrong assertion | 0 | Target zero, and blocking. Nothing was asserted, so nothing could be wrong. Read the note above before using this figure. |
| Correct assertion | 0 | No duty was extracted and adjudicated correct. Not one. |
| Missed | 23 | Every positive label in the set. A refusal where the gold set holds a duty. This is the only figure in the run that carries information: it is the whole of what an extractor would have to earn back. |
| Correct refusal | 28 | Every negative label. Full marks, earned by refusing everything, which is why the metric prints "read it against the extraction count, never alone". |
| Unadjudicated assertion | 0 | Asserted on a positive label with nobody having checked which duty. Nothing was asserted, so this bucket is empty too. |
The 23 misses fall this way by document: 4 in the Georgia 44280 tariff-compliance order, 5 in the Georgia 56002 IRP stipulation order, 1 in the Georgia compliance tariffs, 1 in Kentucky intervenor testimony, 2 in the Missouri ET-2025-0184 order, 1 in its nunc pro tunc order, 2 in the amended global stipulation, 1 in the Missouri compliance tariff revision, 3 in the Utah settlement order, 2 in the Utah settlement stipulation, 1 in the Utah large-load application. Six of the 23 carry the labeller's ambiguous flag.
Four states. The scorer treats the jurisdiction as the sample, four is under the floor of ten, and so no extraction metric on this page prints a rate. The split below is not decoration; it is the sample itemised.
| State | Passages | Source docs | Duties | Hard negatives | Ambiguous | Wrong | Missed | Correct | Refused |
|---|---|---|---|---|---|---|---|---|---|
| Georgia | 19 | 4 | 10 | 9 | 6 | 0 | 10 | 0 | 9 |
| Kentucky | 6 | 4 | 1 | 5 | 3 | 0 | 1 | 0 | 5 |
| Missouri | 13 | 4 | 6 | 7 | 4 | 0 | 6 | 0 | 7 |
| Utah | 13 | 4 | 6 | 7 | 4 | 0 | 6 | 0 | 7 |
| Total | 51 | 16 | 23 | 28 | 17 | 0 | 23 | 0 | 28 |
Georgia is 19 of 51 passages, more than a third, and 10 of the 23 duties. Kentucky holds one duty in six passages, because the Kentucky corpus here is intervenor testimony, errata and cover letters rather than orders. Neither of those is a flaw in the labelling; both mean a result driven by Georgia drafting would look like a result about extraction, and only this table would show the difference.
| Slice | Ambiguous | Of | Share |
|---|---|---|---|
| All labelled passages | 17 | 51 | 33% |
| Positive labels (a duty) | 6 | 23 | 26% |
| Negative labels (no duty) | 11 | 28 | 39% |
A third of the evidence is contestable by the labeller's own note, and the negatives are the more contestable half. The flag means one annotator thought another analyst could argue the other way. It is not a measured disagreement rate. There was one annotator per set, no second pass and no adjudication, so no inter-annotator agreement number exists and none can be computed from these files.
| Metric | Result | Blocking | Verdict |
|---|---|---|---|
| 1. Wrong assertions | 51 of 51 labelled passages carry no wrong assertion. No rate: n = 4 jurisdictions. | yes | PASS, vacuously |
| 2. Obligations extracted, and the ones missed | 0 of 23 labelled obligations extracted and adjudicated correct. No rate: n = 4 jurisdictions. | no | FAIL, reported only |
| 3. Correct refusals | 28 of 28 hard negatives correctly refused. No rate: n = 4 jurisdictions. | no | PASS, vacuously |
| 4. Gold set offsets | 51 of 51 labelled quotes re-read from their source. 100%, n = 16 source documents. | yes | PASS, and this one means something |
Verdict line: 3 of 4 metrics pass, exit 0, no release blocker. The exit code is worthless on this run and the scorer says so. Metric 4 is the only one that survives the reading: every labelled quote still slices out of its source file byte for byte at the recorded offsets, across 16 documents. Each source file really is independent evidence that the text has not moved, so that percentage is the one rate on the page. It says nothing whatever about extraction.
It proves that the scorer runs, refuses correctly, and reports. It refuses to score with no predictions file rather than defaulting to a green page. It refuses a run that is silent about any labelled passage. It refuses an empty gold set. Its own suite passes, 62 tests. And the gold set has not rotted: 51 of 51 quotes re-read.
It does not prove that the extractor does not make false claims, because there is no extractor. That is the first and largest limit, and it is a different sentence from "the extractor made no false claims". Anyone reading "0 wrong assertions" off this page without the next paragraph has been misled by us.
The zero is against this gold set, of this size, labelled by an agent rather than a regulatory expert, over four jurisdictions. 51 passages, 16 documents, 2 files, 4 states. goldset_ga_ky.json names its annotator: "Claude (Opus 5), single annotator, no adjudication". goldset_ut_mo.json names only "human-directed labelling pass, regulatory text only", and its labels were written by hand into a build script sitting in the same directory — so it does not say who made each call, and we should not claim a person did. No licensed regulatory-affairs professional has reviewed either set. Neither is a random sample: hard cases were picked on purpose, so nothing here estimates accuracy over a whole document. The topic range is narrow — large-load service in Utah and Missouri, two Georgia dockets, and a Kentucky corpus of testimony and errata that holds one duty in six passages. Nothing here tests rate design, safety or environmental filings.
And the zero has no partner. Zero correct assertions is on the same page. A scorecard with both halves empty is not a good result and not a bad one; it is a measurement of a thing that has not been built.
shall, must, is required, are required or ordered. It was built in memory, never written to disk, and is not a product path.| Outcome | Keyword baseline | This run |
|---|---|---|
| Wrong assertion | 14 | 0 |
| Missed | 8 | 23 |
| Correct assertion | 0 | 0 |
| Correct refusal | 14 | 28 |
| Unadjudicated assertion | 15 | 0 |
| Exit code | 1 | 0 |
The scorer counted 14 wrong assertions and blocked. So the zero above is a fact about the run, not a metric that cannot fire. It also puts a number on the hard negatives: half of them, 14 of 28, would fool a modal-verb reader. That is the work an extractor has to do, and it is the reason the negatives were chosen.
The scorecard prints a claim the data does not support. Metric 3's method line reads: "Most of them contain the word 'shall'." Measured across the 28 negative labels, 10 contain "shall" — a minority, not most. Widening to any mandatory modal gives 14 of 28, which is exactly half. The sentence is printed on the scorecard where a reviewer meets it, so it should be corrected to the measured figure. app/evals/obligations.py belongs to another lane; this is reported, not changed.
One gold set does not name its labeller. goldset_ut_mo.json carries labelled_by: "human-directed labelling pass, regulatory text only". That phrase reads as though a person made the calls without saying so, and the sibling file is plain that Claude made its calls. The field should say which, in the same words the other file uses. Until it does, this page treats both sets as agent-labelled, because that is the conservative reading and the one we can defend.
Two different things, and they are worth separating. Building an extractor and producing a predictions file would fill the assertion columns and put the blocking metric under real load. Growing the gold set, adding jurisdictions past four, and having a regulatory-affairs professional relabel a sample would make any number computed over it worth more. The first is a build task. The second is the one that decides whether a future zero means anything, and it cannot be done by us alone.
.venv/bin/python -m app.evals.obligations --no-extractor prints every figure on this page. .venv/bin/python -m app.evals.obligations prints the refusal. .venv/bin/python -m pytest tests/test_obligation_evals.py -q runs the scorer's own 62 tests. None of the three touches the network.