WDecision ArchiveEvidence before outcome

Agent evaluation protocol

A case is useful only if the agent cannot peek.

Freeze the evidence, capture the decision, score the process, then reveal the outcome.

Evaluation sequence
  1. LoadPart A only
  2. DecideRecord output
  3. ScoreProcess first
  4. RevealPart B
Public cases are source-bounded, not contamination-resistant.

Named historical cases may already exist in a model's training data. Use neutralized private exports for rigorous benchmark claims.

This protocol turns the historical library into a test and retrieval system for financial analysis. It scores the quality of a decision process before revealing the outcome. It must not reward an agent merely for remembering what happened to a famous company.

The executable reference is warren.case_studies.evaluation. Its current contamination mode is named_case_source_bounded: it enforces the Part A/Part B boundary but does not claim that a model lacks memorized knowledge of a named case.

Two distinct uses

1. Source-bounded decision rehearsal

For a manual rehearsal, give the agent only one frozen part-a/ directory. Do not provide catalog.yaml, episode.yaml, qc.json, the parent directory name, Part B, the corpus index, or a prompt that identifies the outcome class. Require every material assertion to resolve to the Part A source, evidence, fact, claim, assumption, judgment, conflict, or table IDs.

This mode tests whether the agent obeys an information cutoff and reasons from the packet. It does not eliminate knowledge already present in a model's training data.

This is the mode implemented by the deterministic harness. It does not hand the adapter a directory. After freeze and QC verification, it constructs a single in-memory prompt containing only the fixed instructions, response schema, rubric, and canonical Part A JSON. The adapter contract accepts only those prompt bytes; the built-in adapter has an empty read-only filesystem, no tools, and no network.

2. Contamination-resistant prospective evaluation

For a true blind evaluation, a human-controlled export must replace company, person, product, security, regulator, publication, accession, URI, and other recognisable names with stable neutral aliases. Dates may also need a consistent offset while preserving ordering, durations, fiscal periods, and public-availability relationships. Hash and retain the alias map outside the agent context.

The current canonical corpus is intentionally source-authentic and is not itself a de-identified benchmark. A famous company name can activate memorized outcome knowledge even when Part B is physically absent. Do not describe a named-case run as contamination-free.

Alias export is not part of the current harness. A future implementation must bind the alias-map digest and transformed packet digest before selection and must add frozen regressions proving that time, lineage, and cross-reference semantics remain unchanged.

Precommit contract

The evaluation starts from an instructor-side PairPrecommit; evaluation does not select cases opportunistically. It fixes an ordered, reciprocally declared pair with one favorable and one unfavorable case and seals the catalog bytes, selection rule, pairing basis, each episode, freeze, report, and canonical Part A digest, response schema, prompt template, rubric, verification mode, accepted warning codes, and calibration-mapping digest. The role labels remain instructor-side and are never included in the analyst prompt.

The calibration mapping must exist no later than the precommit. For each case it binds the canonical Part A and Part B digests and maps one packet scenario ID to the realized branch. This prevents choosing or rewriting a realized branch after seeing the agent's probabilities. The mapping itself is not opened during the ex-ante phase.

Two verification modes are available and the precommit, not a runtime convenience flag, selects between them:

  • frozen_bundle (default) validates canonical Part A freeze and report digests, cutoff and freeze-time safety, case and rendered-reference QC, and the exact precommitted warning allowlist. It does not claim artifact-byte replay.
  • artifact_backed performs those checks against an existing immutable evidence store as well. The store is required during both evaluation and reveal and is opened read-only by intent; absence or mismatch blocks the phase.

Evaluation sequence

  1. Precommit the pair and mapping. Before either analyst prompt, fix the sampling rule, ordered favorable/unfavorable pair, verification mode, episode/freeze/ report/canonical Part A digests, prompt/schema/rubric digests, and calibration-mapping digest.
  2. Verify both freezes. For each case, use Warren's strict Part A loader, confirm the canonical and report digests, enforce authoring and knowledge times, and run machine and rendered-reference QC in the committed verification mode. When a citation-only report correction is present, also verify the original frozen report bytes and the loader-proven mechanical source-ID rebind; never treat the correction time as a new outcome-blind freeze.
  3. Create the bounded prompt. Supply the adapter only canonical verified Part A, the response schema, and the scoring rubric. Do not expose paths or instructor records. The production adapter disables repository browsing, tools, and network.
  4. Require and bind a decision record. Validate the strict response schema and bind the prompt, transmitted prompt, runtime schema, raw response, canonical response, adapter configuration, and model/runtime coordinates with digests.
  5. Score ex ante quality first. Score evidence use, numerical integrity, uncertainty, alternatives, reversibility, and falsifiability using Part A only. Persist the complete canonical ex-ante pair and its detached Ed25519 OAuth attestation before a production reveal can begin. The run and sidecar are preflighted, staged with restrictive modes, and committed as a fail-closed pair.
  6. Reverify, then reveal. Verify the detached signature from the independently pinned workspace authority, then replay all precommit, Part A, prompt, config, response, and score bindings for both cases. Any blocker prevents loading the mapping or Part B. Then verify the predeclared mapping and Part B digests and record calibration separately.
  7. Extract candidate rules. A lesson remains a candidate until the generated rules/index.json shows the required structure and a human cross-case review confirms independent support, a counterexample, calibration, and acceptable false positives.

Required agent decision record

The response should contain:

  • decision owner, time, knowledge cutoff, and exact question;
  • selected alternative and the strongest feasible rejected alternative;
  • facts, company claims, third-party estimates, assumptions, judgments, conflicts, and unknowns kept in separate lists;
  • financial reconstruction with period, unit, currency, entity, scope, reporting vintage, and source IDs for every material number;
  • business mechanism and the unit at which its economics should be tested;
  • liquidity and capitalization after restrictions, entity location, covenants, collateral, settlement time, and stressed uses—not just headline cash or capacity;
  • at least three scenarios whose probabilities sum to one, with assumptions and practical falsifiers;
  • strongest disconfirming evidence against the selected alternative;
  • missing evidence ranked by expected decision value;
  • reversible and irreversible commitments separated explicitly;
  • leading indicators with definition, source, threshold, cadence, and responsible owner;
  • reversal or kill conditions and the time allowed to obtain evidence;
  • confidence and a statement of what remains unquantifiable; and
  • an explicit valuation abstention: the analyst adapter has no authoritative calculation authority, and missing earnings, cash flow, capitalization, or deterministic inputs independently requires the same fail-closed result.

Ex ante scoring rubric

Score the response before Part B is visible.

The deterministic scorer checks response structure, exact decision coordinates, Part A reference closure, required citations, financial coordinates, scenario and alternative IDs, explicit uncertainty, operational gates, and valuation abstention. It cannot prove semantic entailment, so every result carries a substantive-human- review requirement. The following defects are non-compensating reveal blockers: DECISION_COORDINATES_MISMATCH, UNKNOWN_PART_A_REFERENCE, MATERIAL_ASSERTION_UNCITED, ALTERNATIVE_REFERENCE_INVALID, and SCENARIO_MAPPING_INVALID, plus FINANCIAL_VALUE_BINDING_INVALID, FINANCIAL_VINTAGE_AFTER_CUTOFF, FINANCIAL_PERIOD_AFTER_CUTOFF, EPISTEMIC_REFERENCE_INVALID, SOURCE_LINEAGE_INCOMPLETE, and REQUIRED_DECISION_RECORD_INCOMPLETE, and VALUATION_ABSTENTION_REQUIRED are also blockers: a model-controlled materiality flag, a wrong-kind Part A ID, or placeholder prose cannot satisfy the record contract. A high numerical total never overrides a blocker. A reported value must reproduce its exact Part A fact value, definition, period, unit, currency, entity, scope, and knowledge vintage. A model output is accepted only with live release, execution-replay, and frozen-suite validation authority; a deserialized authoritative flag is insufficient.

DimensionWeightFull-credit standard
Evidence and cutoff discipline15All material assertions resolve to cutoff-valid Part A records; no outside or hindsight facts appear.
Financial integrity15Numbers preserve source, period, unit, currency, entity, definition, and reporting vintage; unknown is never zero.
Epistemic discipline10Facts, claims, estimates, assumptions, judgments, conflicts, and missing evidence remain distinct.
Mechanism and decision unit10The operating mechanism is explicit and economics are tested at the smallest decision-relevant unit.
Alternatives and disconfirmation10Alternatives are genuinely feasible and the strongest opposing evidence is analyzed fairly.
Liquidity and downside10Accessible liquidity and claims are reconstructed under restrictions, timing, covenants, and stress.
Scenarios and calibration10Probabilities sum to one, assumptions are visible, and severe cases are not hidden in a narrative average.
Reversibility and staged commitment10Irreversible capital is gated; immediate-protection exceptions are recognized.
Indicators and kill rules5Thresholds, cadence, evidence owner, and automatic protective actions are operational.
Falsifiability and abstention5The recommendation can be reversed by named evidence, and false precision is refused.

Do not include “matched the eventual outcome” in this score.

The canonical ex-ante output has phase ex_ante_scored and part_b_released: false. It persists the precommit and execution configuration, their digests, per-case verification record, prompt digest, raw and canonical response digests, adapter trace, score, and score digest. Persistence creates a new canonical JSON file and adjacent detached attestation and refuses to replace an existing path. A signature can be issued only from the exact live pair of OAuth trace capabilities; no serialized boolean grants authority. Reveal accepts that persisted form only and verifies it from the independently pinned authority root; an in-memory, unattested, or noncanonical reconstruction is insufficient.

Post-reveal assessment

After the Part A response is frozen, compare it with Part B on separate axes:

  • Signal recognition: Did the response identify the ex-ante signals later connected to the mechanism, without overstating them?
  • False-positive control: Did it preserve favorable evidence and credible benign explanations?
  • Calibration: Did the realized branch receive a nontrivial probability? A qualitative branch can be scored only if the mapping was predeclared; do not invent a branch after seeing the outcome.
  • Process quality: Would the decision still be defensible under another plausible outcome?
  • Protective timing: Were evidence gates and kill conditions early enough for the decision's actual speed of deterioration?
  • Causal discipline: Did the response distinguish structural vulnerability, trigger, amplification, response, external shocks, and later choices?
  • Transfer boundary: Which facts make the lesson applicable or inapplicable to a new decision?

Realized return may be reported, if available and correctly reconstructed, but it must remain separate from process quality.

The harness currently computes deterministic calibration from the predeclared mapping: the probability assigned to the realized scenario and the multiclass Brier score. It also records Part B outcome IDs, observable-signal IDs, and the teaching note's process-quality text. The post-reveal record has phase post_reveal_calibration, binds the ex-ante-run, calibration-mapping, Part B, and ex-ante-score digests, and sets calibration_included_in_ex_ante_total: false and combined_score_prohibited: true. Never average, add, or otherwise collapse the ex-ante process score and outcome calibration into one score.

CLI workflow

The commands are read-only by default. case-studies-evaluate performs the two bounded analyst turns and prints its result without creating an evaluation file unless --write-run is supplied. case-studies-reveal always requires that persisted ex-ante file; it does not create a calibration file unless --write-calibration is supplied. Their JSON summaries expose the phase, evaluation identity, per-case results and relevant digests; evaluation includes eligibility and blocker codes, while reveal includes realized probabilities and Brier scores but no combined score.

uv run warren case-studies-evaluate case_studies \
  --precommit path/to/pair-precommit.json

uv run warren case-studies-evaluate case_studies \
  --precommit path/to/pair-precommit.json \
  --workspace . \
  --write-run artifacts/case-study-evaluation.ex-ante.json

uv run warren case-studies-reveal case_studies \
  artifacts/case-study-evaluation.ex-ante.json \
  --calibration-mapping path/to/calibration-mapping.json \
  --workspace . \
  --write-calibration artifacts/case-study-evaluation.post-reveal.json

For an artifact_backed precommit, pass the same existing store with --artifact-store PATH to evaluation and reveal. Use the same --workspace PATH for persistence and reveal: it controls the bounded Codex runtime and selects the independently pinned, external OAuth authority; it is not made visible to the analyst. No CLI option bypasses attestation. --revealed-at accepts an explicit offset-aware reveal time for reproducible orchestration; it cannot precede ex-ante finalization or exceed the trusted clock observed by the reveal process.

The frozen Netflix/Blockbuster cases in tests/unit/test_case_study_evaluation.py exercise Part-A-only precommit sealing, fixed pair order, verified-only prompt construction, deterministic adapter and score replay, read-only and immutable persistence paths, tamper failures, mapping and Part B binding, and separate Brier calculation. Add or update a point-in-time regression for every material defect in this boundary.

Retrieval protocol for a live decision

Historical cases do not replace current primary research. Use them to generate tests, questions, and failure modes.

  1. Describe the live decision without company names: commitment type, reversibility, funding, legal entities, operating unit, customer harm, evidence gaps, and time to failure.
  2. Match on mechanism and decision archetype, not on industry label or desired outcome.
  3. Retrieve at least one success, one failure, and one boundary or counterexample case.
  4. Build a comparison matrix with columns for similarity, material difference, proposed transfer, falsifier, and missing live evidence.
  5. Read the selected Part A packets before their Part B notes. Make the live decision from current primary evidence; use the historical outcomes only to stress-test the process.
  6. Treat every rule card as a hypothesis unless rules/index.json and human review mark it corpus_validated.

Prompt template

You are evaluating a decision using only the attached frozen Part A packet. Do not use
outside knowledge or infer the historical outcome. Every material assertion and number
must cite a structured Part A ID. Separate facts, company claims, third-party estimates,
assumptions, judgments, conflicts, and unknowns. Reconstruct liquidity and financial
capacity with exact scopes and reporting vintages. Compare all feasible alternatives,
state the strongest disconfirming evidence, assign scenario probabilities summing to
one, select an action, and define dated verification, reversal, and kill gates. Always
abstain from valuation in the analyst response; authoritative valuation belongs only to
deterministic Python with complete governed inputs. Return the required agent decision
record.

Governance

  • Store the precommit and calibration-mapping digests, Part A digest, prompt and transmitted-prompt digests, model/runtime and isolation configuration, raw and canonical response digests, deterministic score and scorer identity, ex-ante finalization time, and separate reveal time.
  • A human must review source entailment and rule promotion; a model cannot approve its own evidence or publication.
  • Do not tune the prompt or scoring rubric on the same cases used for the final evaluation.
  • Hold out cases or use time-separated versions when measuring improvement.
  • Report named-case memory contamination, missing primary records, and unverifiable PDF excerpts as limitations rather than silently accepting them.