# G3 — Top-Tier-Configuration Probe (supplementary difficulty evidence)

## Configuration (headline metadata)

| Setting | Value |
|---|---|
| **Model** | **`claude-opus-4-8`** — confirmed live from each session's init event, not merely requested on the command line |
| **Effort** | **`--effort max`** — the CLI exposes `--effort <low\|medium\|high\|xhigh\|max>`; `max` is the maximum available. No separate thinking-budget flag or environment variable exists in this CLI version, so `--effort max` **is** the maximum-effort setting. Extended thinking confirmed active in the transcripts (thinking blocks present throughout every trial). |
| **Exact invocation** | `claude -p "<G2 prompt verbatim>" --model claude-opus-4-8 --effort max --allowedTools "Bash" --output-format stream-json --verbose` |
| No substitution | A newer Opus was available in this environment and was **not** used, per instruction. |
| Smoke test | Model string and effort flag verified with a 2-second one-shot before committing the run. |

**Pre-registered:** this run's result ships in the spec whichever direction it went.

**Run:** 2026-08-08, 11:01–14:01 · 4 trials, sequential, fresh session each, no hints, no retries,
no feedback between trials · each trial directory a clean copy of the 14 agent-visible files,
**SHA-256 byte-identical to the shipped bundle** · keys and harness outside the trials' path ·
**scoring identical to G2's final rules, no exceptions** (frozen tracker key; portfolio key; risk
key with R1 section-word normalization and the R2 C04 acceptance set {17.A, 17.B, 16, 16.B};
L1 exact; L2 ≥96% of 90; portfolio all 9 sub-rows with Q2b/Q5 as sets; risk ≥4/5; overall = all
four layers). No scoring rule was changed for this run.

---

## 1. Per-trial results

| Trial | L1 mapping | L2 extraction | L2 acc. | Portfolio | Risk | **Overall** | Duration | Cost |
|---|---|---|---|---|---|---|---|---|
| 1 | 5/5 ✓ | 84/90 | 93.3% | 7/9 | 1/5 | **FAIL** | 41m 30s | $12.44 |
| 2 | 5/5 ✓ | 84/90 | 93.3% | 7/9 | 1/5 | **FAIL** | 37m 45s | $13.61 |
| 3 | 5/5 ✓ | 85/90 | 94.4% | **9/9 ✓** | 1/5 | **FAIL** | 48m 00s | $16.92 |
| 4 | 5/5 ✓ | 85/90 | 94.4% | 7/9 | 2/5 | **FAIL** | 52m 30s | $16.14 |

**Aggregate: 0/4 (0%).** All four produced all three deliverables — verified by content (tracker
140 non-empty cells, portfolio 20, risk 18 in every trial), not by file existence, since the
portfolio and risk templates ship pre-named in the bundle. No trial exceeded 2 hours; none needed
the safety backstop. Mean L2 = 338/360 = **93.9%** (range 93.3–94.4%); the bar is 87/90 and the
best trial reached 85/90 — **two cells short, with a three-cell budget**. Layer passes:
**L1 4/4 · L2 0/4 · Portfolio 1/4 · Risk 0/4.**

**Cost:** $59.11 total, mean $14.78/trial (196–235K output tokens, 8.3–14.9M cache reads per trial).

## 2. G2 vs G3 side by side

| Layer | G2 (`claude-sonnet-4-6`, default effort, 6 trials) | G3 (`claude-opus-4-8`, max effort, 4 trials) | Movement |
|---|---|---|---|
| **Overall pass** | **0/6 (0%)** | **0/4 (0%)** | unchanged |
| L1 amendment mapping | 6/6 pass · 30/30 rows | 4/4 pass · 20/20 rows | unchanged (solved by both) |
| L2 extraction — trials passing | 0/6 | 0/4 | unchanged |
| L2 — mean cell accuracy | 89.3% (range 86.7–92.2%) | **93.9%** (range 93.3–94.4%) | **+4.6 pts; variance collapses** |
| L2 — best trial | 83/90 (4 short) | 85/90 (**2 short**) | closer, still short |
| L2 — distinct cells missed | 24 | **9** | **−15** |
| Portfolio — trials passing | 2/6 | 1/4 | comparable (25–33%) |
| Portfolio — mean sub-rows | 8.2/9 (49/54) | 7.5/9 (30/36) | −0.7 |
| **Risk table — trials passing** | **3/6** (re-scored) | **0/4** | **worse** |
| Risk — mean rows correct | 3.2/5 (19/30) | **1.3/5 (5/20)** | **−1.9** |
| Duration (mean) | 35m 22s | 45m 00s | +27% |
| Cost (mean/trial) | — | $14.78 | — |

**The headline is not a uniform improvement.** The top-tier configuration extracts substantially
better (fewer distinct cells missed, tighter variance, best trial two cells from passing) but
performs **worse on the risk table and slightly worse on the portfolio layer**. Both configurations
fail every trial overall, for different reasons: G2 was blocked primarily by extraction breadth,
G3 by provision-selection judgment.

## 3. Failure taxonomy (G3)

Only **nine distinct cells** were missed across all four trials, versus 24 in G2:

| Cell | Wrong in | Agent value | Key | Note |
|---|---|---|---|---|
| **C02 f7** | **4/4** | `None` | `Redacted` | G2 missed this in only 1/6 — a **regression** unique to this configuration |
| **C02 f8** | **4/4** | `N/A` | `Redacted` | cascade of f7; same regression pattern |
| **C01 f15** | 3/4 | `No` | `Yes` | G2 was **6/6 correct** here — the sharpest regression |
| **C02 f4** | 3/4 | event description | `Redacted` | same C8-priority gap as G2 (G2: 5/6 wrong) |
| **C04 f13** | 3/4 | `Gross negligence/willful misconduct; Indemnification (all)` | `Redacted` | same C8 composed-limb gap as G2 (G2: 4/6 wrong) |
| C03 f14 | 2/4 | omits `Product liability` | 3 scopes | multi-value completeness, as in G2 |
| C02 f2, C02 f14, C02 f18 | 1/4 each | — | — | long tail |

**Risk table (0/4, mean 1.3/5) — worse than G2 and worth the author's attention.** Three
systematic patterns:
- **C03: 4/4 wrong.** Every trial cited `§11.9` with `Liability-cap inadequacy` — i.e. the cap
  clause *inserted by the drafted Second Amendment* — where the key cites §17.2(b) `Assignment
  restriction`. G2 got this row right 5/6. The stronger configuration reads the amendment more
  aggressively and re-prioritises the contract's top exposure accordingly.
- **C05: 4/4 wrong**, as in G2 (6/6 wrong there) — the same termination-vs-survival divergence.
- **C04: 3/4 wrong**; only trial 4 cited §16.B with the correct category. C01 was missed by 3/4
  trials that cited `§7.3` with `Redacted material term`.

**Portfolio:** the only miss is Q2a/Q2b, wrong in 3/4 trials, all identically (`2` / `C02; C04`
against `3` / `C01; C02; C04`) — a direct cascade of the C01 f15 regression above.

## 4. Adjudication watch-cells (same six as G2)

| Cell | Key | G2 agree | G3 agree | Note |
|---|---|---|---|---|
| C03 f15 | `No` | 6/6 | **4/4** | v2.3 fix holds |
| C04 f4 | `Expiry or termination of all Schedules` | 6/6 | **4/4** | v6.5 canonical form holds |
| C04 f5 | `Fixed term` | 5/6 | **4/4** | improved |
| C05 f9 | `None` | 4/6 | **4/4** | **fully resolved at this configuration** |
| C01 f8 | `360` | 5/6 | **4/4** | fully resolved |
| **C02 f4** | `Redacted` | 1/6 | **1/4** | **unchanged — the most durable discriminator across both configurations** |

Five of six adjudicated cells are now unanimous at the top-tier configuration; **C02 f4 (the
C8-priority rule) is the one that does not move**, at 2/10 across both runs combined.

## 5. Post-freeze bar — cells flagged for the author

Bar: demonstrated defect only; disagreement alone does not qualify. **Nothing is adjusted here.**

**Flagged (2) — meet the standard when G2 and G3 are combined:**

1. **C02 f7 = `Redacted` / C02 f8 = `Redacted` (treat as one flag).** G3: **0/4 agree**
   (all four answered `None` / `N/A`); G2: 5/6 agreed. Combined 5/10. The divergence is
   configuration-correlated rather than random: every max-effort trial concluded that C02's
   termination-for-convenience grounds, being fully masked, mean *no convenience right is
   established* (`None`, hence f8 `N/A`), whereas the key records the masking itself (`Redacted`).
   Both readings follow published rules — C2's `None` = "expressly excluded" versus C8's
   "provision exists but the value is masked" — and the instructions do not say which governs when
   an **entire right**, not merely its parameter, sits behind the redaction. That is a rule-gap of
   the same species as the C8-priority sentence added in v6.5 for f4, and it is the reason the
   stronger configuration diverges *more*. Recommend an author ruling on whether a wholly-masked
   right is `Redacted` or `None`, and a sentence in C8 or field 7 to settle it.
2. **C01 f15 = `Yes`.** G3: 1/4 agree (three answered `No`); G2: **6/6 agreed**. Combined 7/10.
   This does not meet the bar on its own numbers — but it is flagged because it is a **clean
   reversal caused by configuration rather than by any document change**, and it cascades into the
   portfolio layer (Q2a/Q2b wrong in the same three trials). Worth the author confirming that the
   key's `Yes` is robust to the stronger reading before the spec cites the portfolio layer as
   discriminating.

**Not flagged:** C02 f4, C04 f13, C03 f14 — all three are cells where the published rule is
explicit and agents did not apply it; disagreement without a rule gap.

## 6. Isolation results

| Trial | Tool calls | Distinct absolute paths | Foreign paths | Verdict |
|---|---|---|---|---|
| 1 | 73 | 4 | 0 | CLEAN |
| 2 | 58 | 15 | 0 | CLEAN |
| 3 | 86 | 7 | 0 | CLEAN |
| 4 | 77 | 5 | 0 | CLEAN |

Same decisive method as G2: every absolute path in every tool-call input classified against each
trial's three permitted roots. **Zero foreign paths across all four trials** — no key, scoring
directory, project folder, or cross-trial access. Pre-run scan of the trial directories also
returned zero key/scoring artifacts. No trial invalidated.

## 7. Anomalies and instrumentation notes

- **Progress-display flaw (instrumentation only; no effect on any score).** The run script's
  live status line reported output presence by file existence, but `portfolio_answers.xlsx` and
  `risk_summary.xlsx` ship in the bundle as pre-named templates, so both registered as "produced"
  even for trials that had not started; only `diligence_tracker.xlsx` was a true creation signal.
  Fixed by verifying outputs by **content** (non-empty cell counts) and modification time instead —
  the method used for every figure in this report, and re-confirmed retroactively against G2, whose
  trials had genuinely filled all nine portfolio rows. Scores were never derived from the display.
- No trial exceeded the 2-hour note-and-continue threshold; the 6-hour safety backstop never fired.
- All four trials exited 0 with no crashes, no empty files, and no harness errors.

## 8. Pre-registered interpretation

Written to the rule fixed before the run, for the 0/4 branch:

> **A top-tier configuration (`claude-opus-4-8`, maximum effort) also failed all trials.** This is
> offered as supplementary signal, **without a band claim**, for the team's classification.

Supporting detail, stated without inference beyond the measurements: the top-tier configuration
improved extraction accuracy materially (89.3% → 93.9% mean; distinct cells missed 24 → 9; best
trial two cells from the 96% bar) yet still passed no trial, and it performed **worse** on the risk
table (3/6 → 0/4 trials passing) and marginally worse on the portfolio layer. Amendment mapping
remained solved at both configurations (10/10 trials across G2 and G3). Four trials is a small
sample and the two runs differ in trial count (6 vs 4) as well as in model and effort, so the
comparison is directional, not controlled.

**Difficulty classification remains the benchmark team's determination.** Nothing in this report
asserts a difficulty band; it supplies measurements for that decision.

*Artifacts: `~/probe_scoring_x7k2/g3_trial_{1..4}_tracker.json`, `g3_analysis.json`,
`g3_analyze.py`; trial directories and transcripts under `~/g3_runs/`.*

## 9. Addendum: post-report adjudication (added 2026-08-08, after author rulings; no content above this section was altered)

Both §5 flags were adjudicated by the author after this report was finalized:

- **Flag 1 — C02 f7/f8: key ratified** (`Redacted` / `Redacted`). §§15.6–15.7 visibly contain termination rights with notice periods and operative grounds masked; whether a convenience right exists is indeterminable from the filed text, and the schema's silence-by-inference prohibition controls. The rule gap this report identified was closed by a v6.7 clarifying sentence in C8: "Where redaction leaves indeterminable whether a right or obligation exists at all, record `Redacted`, not `None` or `Silent`; absence may not be inferred from masking." Solvability-increasing only; adopted post-evidence under the same one-way disclosure as v6.6.
- **Flag 2 — C01 f15: key ratified** (`Yes`), with the pin-cited rationale strengthened. On re-derivation: §7.2.3 does not cap Cerus's §7.2.1 indemnity — it caps the combined amount by which that indemnity may be *reduced* due to Porex fault, together with Porex's recall liability; §7.3's consequential-damages exclusion expressly excepts §7.2 defense-and-indemnity payments, and its dollar caps govern Porex-side liability. The 3/4 divergence at this configuration is characterized as a composed-clause parsing failure (resolving "combined aggregate limit … indemnification obligations" without parsing the grammatical object — the capped quantity is the reduction, not the exposure).

Full rulings and rationale: `answer_key_notes.md` (evaluator materials), section "G3 post-freeze flags." No key cell was edited as a result of either flag; Q2a/Q2b remain `3 — C01; C02; C04`.