One request, sent 2026-08-05 at 15:43 UTC (08:43 PDT) through app/interpretation/propose.py's AnthropicTransport against claude-opus-5. Command: .venv/bin/python scripts/probe_live_transport.py --send.
--send believing the key was absent because it had cleared ANTHROPIC_API_KEY from the environment. load_env() puts any name from .env back that is not already set, which is exactly what clearing it creates, so the key was present and a second request almost certainly went out. Its output was piped to head -2 and lost, so nothing about it can be reported: not the response, not the token counts, not whether it succeeded. Everything below describes the first, deliberate call. The second is named here rather than left out because a page that quietly says "once" while its author knows of a second call is the exact failure this file exists to correct. scripts/probe_live_transport.py now warns in its own header that clearing the variable does not disarm it.make test passes with no key and no network — and it left the class that actually talks to the endpoint as documented design that had never run. Two module docstrings said so honestly, and honesty does not make a request legal: a parameter invented from memory, a response field read from the wrong place, or a combination the endpoint rejects outright all pass a fake and all fail once. This is that once, written down.| Question | Answer |
|---|---|
| Date | 2026-08-05, 15:43:37 UTC |
| Script | scripts/probe_live_transport.py, run with --send. Without that flag it prints the request the transport builds, needs no key, and sends nothing. |
| Transport | app/interpretation/propose.py — the citation-gate path: no tools, structured output constrained to the judgement schema, adaptive thinking. |
| Transport NOT exercised | app/chat/agent.py — a second AnthropicTransport with a different parameter set (tools, and effort inside output_config). It has still never run, and its docstrings still say so. |
| Model requested | MODEL_ID, read from the module rather than typed into the script. |
| Model the endpoint reported | The same id, echoed on response.model. |
| Key | Loaded from .env by app.config.load_env(), exactly as the application loads it. The script prints its length and never its value. |
| Retries | Bounded to one, set explicitly rather than left at the client default of two. |
Printed by the script, and regenerable on any machine with no key and no charge: .venv/bin/python scripts/probe_live_transport.py drives the shipped transport against a client that records the request and then refuses to send it, and prints exactly the block below. The schema is the module's _SCHEMA; the prompt is the script's and is deliberately tiny.
tests/test_live_transport_probe.py now rebuilds the request through AnthropicTransport and holds this block to the exact bytes, so it cannot be edited at all — and it fires on any later drift in the model id, the token ceiling, the schema or the prompt.{
"model": "claude-opus-5",
"max_tokens": 16000,
"system": "You judge whether one change to a regulatory filing matters to the utility that has to live with it. Answer with one object: material, true or false; why, in one sentence; and a citation naming the version, the 0-based character offsets of the exact span, and the text at that span copied character for character.",
"thinking": {
"type": "adaptive"
},
"output_config": {
"format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"material": {
"type": "boolean"
},
"why": {
"type": "string"
},
"citation": {
"type": "object",
"properties": {
"version_id": {
"type": "string"
},
"char_start": {
"type": "integer"
},
"char_end": {
"type": "integer"
},
"quoted_text": {
"type": "string"
}
},
"required": [
"version_id",
"char_start",
"char_end",
"quoted_text"
],
"additionalProperties": false
}
},
"required": [
"material",
"why",
"citation"
],
"additionalProperties": false
}
}
},
"messages": [
{
"role": "user",
"content": "VERSION v-probe-1, whole text:\nUtilities shall file the annual reliability report by March 1.\n\nCHANGE: the deadline moved from April 1 to March 1."
}
]
}| Field | Value |
|---|---|
response.model | claude-opus-5 |
response.id | msg_011CdjrjuHCmhR9TQR3TaBap |
response.stop_reason | end_turn |
| Content block types | ['thinking', 'text'] |
usage.input_tokens | 523 |
usage.output_tokens | 453 |
usage.cache_creation_input_tokens | 0 |
usage.cache_read_input_tokens | 0 |
The string complete() returned, which is the text blocks joined and nothing else:
{"material": true, "why": "Advancing the annual reliability report deadline by a full month compresses the utility's data-collection, review, and approval timeline for a mandatory filing, so it changes an enforceable compliance obligation.", "citation": {"version_id": "v-probe-1", "char_start": 54, "char_end": 61, "quoted_text": "March 1"}}| Claim | Status | Evidence in the transcript |
|---|---|---|
| The key loads the way the application loads it | Confirmed | A request authenticated with what load_env() put in the process. No key was exported by hand. |
| The endpoint accepts this parameter combination | Confirmed | A 200 with stop_reason: end_turn. thinking: {"type": "adaptive"} and output_config.format were both accepted, and no sampling parameter was sent. |
| The model id is real and is the one that answered | Confirmed | response.model echoes what the module pins. |
| Structured output returns the shape the schema asks for | Confirmed once | The text parses as JSON and carries material, why and a four-field citation. |
| The parser reads the right field | Confirmed | A thinking block came back first, so joining only the text blocks is required rather than tidy. An earlier version of this row gave the wrong reason for that — it said indexing content[0] would have returned the reasoning. It would have raised AttributeError: the SDK's ThinkingBlock carries signature, thinking and type, and no text field at all. There was also no reasoning in it to return. This model never returns a raw chain of thought, and the request sends no display, whose default is omitted, so the block's thinking field is an empty string. |
| The refusal check sits before the content read | Not exercised | Nothing declined. The branch was not taken and remains fake-tested only. |
| Offsets are trustworthy | Not settled | The offsets returned — 54 to 61 — do land exactly on March 1 in a 62-character single line. That is one span in one sentence, and the verifier re-reads every one of them anyway, which is the design and not an accident. |
It proves the wire format, the authentication and the parse work once, on a prompt of 523 input tokens, on a quiet Wednesday morning, from one machine. It proves nothing about behaviour under load: no rate limit was hit, so the client's retry and back-off have still never run. It proves nothing about retries, timeouts or a connection dropped mid-response, because none happened. It proves nothing about long inputs — a real change carries two passages from a filing and a company context, and both the token count and the model's accuracy at counting characters over thousands of them are untested here. It proves nothing about the citation gate, which is the part this product rests on: the verifier was never invoked, no judgement was stored, and a citation that fails to verify was never produced by a real model, only by a fake that was told to produce one. It proves nothing about app/chat/agent.py's AnthropicTransport, which is a second class with a different parameter set — tools, and effort inside output_config — and which has still never run. One call is the difference between a design and a fact, and it is a small difference. It moves the model path from unexercised to exercised, and no further.
tests/test_live_transport_probe.py holds five separate claims on this page to something outside it. It reads MODEL_ID out of both model modules: the marked id above must equal what the application pins, and no other model id may appear anywhere on the page. Change the model and the suite fails, naming this file — because a transcript that describes a model nobody calls reads exactly like a transcript that describes the model everybody calls, and nothing on the page would otherwise tell a reader which it is.
Three more were added after a reviewer found the first version of this page wrong in three places. The request block is rebuilt by running the shipped transport and compared byte for byte, so it cannot be tidied, retyped or left behind by a change to the schema or the token ceiling. The two module names in the table above are read off this page and their docstrings are checked against it — the one marked exercised may not still say it has never run, and the one marked unexercised must still say that it has. Both directions matter: the first is the sentence a correction leaves behind, the second is the caveat a tidy-up deletes. The thinking-block reason is checked against the installed SDK's own field list rather than against memory.
What no test on this page can check remains large, and is worth naming beside the guards: the token counts, the response body, the message id and the date are trusted prose. A test can keep one field of a transcript from rotting. It cannot verify that a call happened.