# G2 — Final Full-Task Trial Report

**Versions under test: schema v6.5 · Amendment_02 v2.3 · answer key FROZEN
(`answer_key_tracker_FROZEN.xlsx`) · 14-file agent-visible bundle.**
*(Instructions have since been cut to v6.6 — see §7 disclosure. All statistics below were measured on v6.5.)*

**Run:** 2026-08-08, 01:54–05:27 · reference agent `claude-sonnet-4-6`, headless (`claude -p`),
default effort, `--allowedTools "Bash"` · six independent trials, sequential, fresh session each,
no hints, no retries, no feedback between trials · each trial directory a clean copy of the
14 agent-visible files only · keys and harness held in `~/probe_scoring_x7k2/`, outside the trials'
path · scoring: `score_task.py` + `scoring_config_N5.json` (L2 ≥96% of 90 cells; L1 exact),
`portfolio_key.xlsx` (9 sub-rows, exact; `Q2b`/`Q5` set-compared), `risk_key.xlsx` (section
prefix-tolerant, category exact, pass ≥4/5) — risk figures reported both as originally scored and
as re-scored under post-freeze rulings **R1** and **R2** (§5).

---

## 1. Per-trial results

| Trial | L1 mapping | L2 extraction | L2 acc. | Portfolio | Risk (orig) | **Risk (re-scored)** | **Overall** | Duration |
|---|---|---|---|---|---|---|---|---|
| 1 | 5/5 ✓ | 78/90 | 86.7% | 8/9 | 3/5 | **3/5** | **FAIL** | 35m 15s |
| 2 | 5/5 ✓ | 82/90 | 91.1% | 9/9 ✓ | 3/5 | **4/5 ✓** | **FAIL** | 28m 00s |
| 3 | 5/5 ✓ | 79/90 | 87.8% | 8/9 | 2/5 | **2/5** | **FAIL** | 31m 00s |
| 4 | 5/5 ✓ | 83/90 | 92.2% | 9/9 ✓ | 3/5 | **4/5 ✓** | **FAIL** | 51m 45s |
| 5 | 5/5 ✓ | 81/90 | 90.0% | 8/9 | 3/5 | **4/5 ✓** | **FAIL** | 34m 30s |
| 6 | 5/5 ✓ | 79/90 | 87.8% | 7/9 | 0/5 | **2/5** | **FAIL** | 31m 45s |

**Aggregate pass rate: 0/6 (0%) — unchanged by the re-scoring.** All six produced all three
deliverables; no auto-fails. Mean L2 = 482/540 = **89.3%** (range 86.7–92.2%; best trial 83/90
against a bar of 87/90 — four cells short with a three-cell budget). Portfolio mean 8.2/9, with
**2/6 trials perfect**. Risk table **14/30 → 19/30** under R1/R2, moving **0 → 3 trials** past the
4/5 bar. Mean duration 35m 22s.

Layer-level pass counts: **L1 6/6 · L2 0/6 · Portfolio 2/6 · Risk 0/6 → 3/6 (re-scored).**
L2 remains the universal blocker: no trial cleared 96% on extraction, so overall pass stays 0/6
under either scoring.

Risk rows passing, re-scored (of 6 trials): **C01 6/6 · C02 5/6 · C03 5/6 · C04 3/6 · C05 0/6.**

## 2. Failure taxonomy (all layers)

### L2 extraction — 24 distinct cells missed, 58 cell-instances

| Cell | Wrong in | Agent value | Key | Character |
|---|---|---|---|---|
| C05 f10 | **6/6** | `Consent w/ affiliate carve-out` | `Consent required` | over-reads §11's technology-use proviso as an assignment carve-out |
| C02 f2 | 5/6 | `ORPHAN EUROPE, S.A.R.L.` | `ORPHAN EUROPE` | preamble-fidelity: agents import the suffix from the signature block |
| C02 f4 | 5/6 | `10 years from Marketing Authorization…` (4 wordings) | `Redacted` | C8-priority rule vs the f4 pattern rule |
| C03 f4 | 5/6 | `Expiry of all Royalty Terms` | `Expiry of all Royalty Terms in all countries` | truncation of the ≤10-word event description |
| C03 f14 | 4/6 | omits `Product liability` | 3 scopes | multi-value completeness |
| C04 f13 | 4/6 | `Gross negligence/willful misconduct; Indemnification (all)` | `Redacted` | C8 composed-limb rule vs visible carve-out list |
| C02 f18 | 3/6 | `No` | `Yes` | expanded-MFN definition missed |
| C04 f14 | 3/6 | adds `Data/privacy` | 3 scopes | over-inclusion |
| 16 further cells | 1–2/6 each | — | — | long tail, no consensus |

### Portfolio — 4 misses across 54 sub-rows (92.6% cell accuracy)
`Q4a` 365 vs 360 (trial 1) · `Q5` `None` vs `C05` (trials 3, 5 — the f9 cascade) ·
`Q2a`/`Q2b` 4 / `C01; C02; C04; C05` vs 3 / `C01; C02; C04` (trial 6 — f15 over-inclusion).
The two perfect trials (2, 4) show the layer is achievable.

### Risk table — re-scored under R1/R2 (0/6 → 3/6 passed)
After the two evaluator rulings, the layer separates into one solved row, two near-solved, and two
genuinely contested:

- **C05: 0/6 — unanimous, and unaffected by either ruling.** Every trial cited a termination
  provision (§8.4.1 / §8.4 / §8.3) with category `Termination exposure`; the key cites §8.4.6 with
  `Liability-cap inadequacy`. Section *and* category diverge in all six. The one trial whose cite
  did match (trial 2, §8.4 prefix-matches §8.4.6) still failed on category. This is the layer's
  hardest row: agents locate the termination architecture, the key locates the survival clause that
  preserves uncapped liability.
- **C04: 0/6 → 3/6 under R2.** Trials split evenly: three cited §16.B with the correct category
  (now matched by the acceptance set), three cited §23.B with `Change-of-control exposure` (still
  a miss on both limbs). The source hazard noted in the cite-verification step is relevant: §17.A's
  own carve-out sentence mis-labels itself "this Section 17.B".
- **C01 6/6 · C02 5/6 · C03 5/6** — the three rows where key and agents converge on the same
  provision. C02's single miss (trial 6, §17.1 + `Exclusivity restraint`) and C03's (trial 3,
  §11.9 + `Liability-cap inadequacy`) are substantive, not formatting.
- **Trial 6's 0/5 was substantially a formatting artifact**, now corrected by R1: it wrote every
  cite as `Section 7.2.1`, which the original normalization (stripping only §/whitespace/case)
  failed on the literal word. Re-scored 2/5 — its three remaining misses are substantive.

### L1 mapping — 6/6, no misses
Every trial resolved all five rows, including the C03 multi-value cell (`A01; A02`) and the
Amendment_03 → C02 attachment with its embedded medac-addendum distractor. Across G1 and G2 the
mapping layer is now 10/10 trials perfect.

## 3. Adjudication verification (did D2–D5 move?)

| Watch cell | Key | Agree with key | G1 baseline | Verdict |
|---|---|---|---|---|
| **C03 f15** (v2.3 carve-out) | `No` | **6/6** | 3/4 answered `Yes` **against** the key | **FIXED — decisive.** Re-scoping carve-out (a) to Novartis-only flipped a 3-of-4 miss into unanimous agreement. |
| **C04 f4** (v6.5 exhaustion form) | `Expiry or termination of all Schedules` | **6/6** | 2/4 phrasing miss | **FIXED — decisive.** The canonical form removed the free-text variance entirely. |
| **C04 f5** (v6.5 subordinate-instrument rule) | `Fixed term` | **5/6** | 2/4 answered `Evergreen` | **FIXED (residual 1/6).** Trial 4 still reads `Evergreen`. |
| **C02 f4** (C8-priority rule) | `Redacted` | **1/6** | 2/4 against key | **NOT FIXED.** Five trials give an event-description in four different wordings; only trial 4 applied C8 priority. The sharpest remaining schema/agent gap. |
| **C05 f9** (unanimous discriminator) | `None` | **4/6** | 0/4 (unanimous `30` against key) | **PARTIALLY MOVED.** Two trials still read `30`. Ten of ten agents said `30` pre-fix; four of six now say `None` — the D1 rule text is landing, imperfectly. |
| **C01 f8** (C4 mechanical table) | `360` | **5/6** | 3/4 answered `365` | **MOVED.** Only trial 1 still reads `365`. |

**Caveat on interpretation:** these are directional comparisons, not a controlled before/after.
G1 measured different materials (Amendment_02 v2.2, schema v6.4, the retired draft key) with four
trials; G2 measures the frozen configuration with six. The magnitudes are informative; the
direction is unambiguous for C03 f15 and C04 f4.

## 4. Consensus-against-key under the post-freeze bar

Bar: **demonstrated defect only — disagreement alone does not qualify.** Eleven cells show ≥2
trials agreeing against the key. Assessed against the shipped documents and the v6.5 rules:

**Qualifying (1) — recommend author review:**
- **C05 f10 (6/6 unanimous).** Every trial reads §11's proviso — "this section shall not prohibit
  the parent of the Licensee, or any subsidiary … from utilizing the Ehave Technology" — as an
  affiliate carve-out. The key's `Consent required` is correct on my reading (the proviso permits
  *technology use*, not assignment), but unanimity across ten agent-readings (4 in G1 + 6 here)
  against a cell whose text is short and unambiguous suggests the *field-10 definition* does not
  tell an agent how to distinguish a use-permission proviso from an assignment carve-out. This is
  a documented instruction gap, not merely disagreement. **Resolved by R3** (§5): the field-10
  sentence added in v6.6 supplies the missing criterion; the key value stands unchanged.

**Not qualifying (10) — disagreement without demonstrated defect:** C02 f2 (suffix import — key
follows the preamble as v6.5 requires), C02 f4 and C04 f13 (agents didn't apply the C8-priority
rule that v6.5 states explicitly), C03 f4 (truncation), C03 f14 / C04 f14 (multi-value
completeness), C02 f18 (definition stated expressly), C01 f6 / C01 f17 (2 trials each), C05 f9
(rule now stated; majority complies).

## 5. Post-freeze rulings applied (R1–R3)

All three were made after the key freeze. **No key cell was edited**; the five risk-key rows are
unchanged. R1 and R2 change the evaluator; R3 changes the agent-facing instructions.

| Ruling | What changed | Layer | Rationale | Measured effect |
|---|---|---|---|---|
| **R1** | Risk-section normalization additionally strips a leading `Section`/`Sec.`/`Clause`/`Cl.`/`Art.` token before the existing §/whitespace/case stripping; prefix tolerance unchanged | evaluator | One trial wrote every cite as `Section 7.2.1` — identifying the right provision but failing on the literal word. The rule was scoring phrasing, not provision identification. Loosening only: it can convert a miss to a match, never the reverse. | trial 6 risk 0/5 → 2/5; no other trial's section results change |
| **R2** | C04 risk-section acceptance set = {`17.A`, `17.B`, `16`, `16.B`}; **category matching stays exact** | evaluator | C04's uncapped-indemnity exposure is locatable at either the cap clause whose carve-out releases it (§17.A) or the indemnity released (§16/§16.B); and §17.A's own carve-out sentence mis-labels itself "this Section 17.B", so the contract's internal cross-reference points at 17.B. All four cites are defensible for the same exposure. The key row still reads `17.A \| Uncapped indemnity exposure` — this is tolerance, not correction. | C04 row 0/6 → 3/6 (the three trials citing §16.B with the correct category); §23.B cites still miss on both limbs |
| **R3** | Instructions **v6.6**, field 10: "A proviso permitting a parent or affiliate to use or exploit rights under the agreement is not an assignment carve-out; carve-outs address the transfer of a party's rights and obligations, not their exercise." | agent-facing | C05 f10 was missed by 10/10 agents across G1+G2, all reading §11's technology-use proviso as an affiliate carve-out. The key (`Consent required`) is correct and **stands**; field 10 simply gave no criterion for distinguishing a use-permission proviso from a transfer carve-out. | not reflected in any statistic below — see §7 |

## 6. Isolation results

| Trial | Tool calls | Distinct absolute paths | Foreign paths | Verdict |
|---|---|---|---|---|
| 1 | 79 | 13 | 0 | CLEAN |
| 2 | 54 | 8 | 0 | CLEAN |
| 3 | 54 | 7 | 0 | CLEAN |
| 4 | 66 | 15 | 0 | CLEAN |
| 5 | 74 | 8 | 0 | CLEAN |
| 6 | 43 | 8 | 0 | CLEAN |

Method: every absolute path appearing in any tool-call input, classified against each trial's three
permitted roots (its own directory, its own sandbox scratchpad, its own session tool-results
directory). **Zero foreign paths across all six trials** — no trial accessed a key, the scoring
directory, the project folder, or another trial. Pre-run leak scan of the six trial directories
also returned zero key/scoring artifacts. No trial invalidated.

## 7. Version disclosure — v6.5 measured, v6.6 shipped

Every statistic in this report was measured against the **v6.5** instructions docx (with
Amendment_02 v2.3 and the frozen key). **R3 ships in v6.6, cut after the trials were scored**, so
no trial saw the new field-10 sentence.

The direction of that gap is one-way. R3 adds a distinguishing criterion where the schema
previously supplied none; it removes no information and imposes no new requirement. A v6.6 re-run
could therefore only improve performance on C05 f10 — the single cell it addresses — and cannot
degrade any other cell. **The reported 0/6 overall pass rate is accordingly an upper bound on
difficulty-as-shipped, never an understatement.** For the same reason, C05 f10 is excluded from
the discriminator claims in §8: its 6/6 miss rate reflects a since-corrected instruction gap, not
task difficulty.

R1 and R2, by contrast, are fully reflected above: they are evaluator-side and were applied to all
six trials, with original and re-scored figures reported side by side in §1 and §2.

## 8. Difficulty evidence (for the final spec)

Under the frozen configuration — **schema v6.5 (shipped as v6.6; see §7), Amendment_02 v2.3,
`answer_key_tracker_FROZEN.xlsx`, and the 14-file agent-visible bundle** — a second-tier reference
agent (`claude-sonnet-4-6`, headless, default effort) **passed 0 of 6 independent full-task
trials**, under both the original and the re-scored evaluator. Every trial produced all three
deliverables and every trial cleared amendment mapping 5/5, so the failures are not artifacts of
format, refusal, or incomplete work: they are extraction and judgment failures on the merits.

**Per-layer aggregates.** Extraction (L2): mean 89.3% cell accuracy against a 96% bar, range
86.7–92.2%; the best trial finished four cells short with a three-cell budget, and **no trial
passed**. Portfolio: mean 8.2/9 with 2/6 trials perfect — the layer is achievable but unforgiving,
since a single wrong sub-row fails it. Risk table: 19/30 rows correct after R1/R2, with 3/6 trials
clearing the 4/5 bar. Amendment mapping: 6/6, the one layer the reference agent solves reliably.
Extraction is the universal blocker; the risk table is the second constraint.

**Published-rule discriminators.** Two cells discriminate specifically on whether an agent applies
a published rule over its own prior: **C05 f9** (the required-condition cure rule) held at 4/6
agreement — the rule is landing on a majority but two trials still record the stated 30-day window
— and **C02 f4** (the C8-priority rule) held at only **1/6**, five trials preferring an
event-description in four different wordings over the mandated `Redacted`. Both cells are correctly
keyed, both rules are stated in the shipped instructions, and both remain live discriminators.
C05 f10 is **not** claimed as a discriminator: its 6/6 miss rate reflected a genuine instruction
gap, now closed by R3.

**Caveat on the mapping layer.** L1's 6/6 (10/10 across G1 and G2) should not be read as validating
amendment mapping as a difficulty source at this scale. With three amendments against five
candidate parents — one multi-value cell and one embedded distractor — the layer is small enough
that the reference agent solves it consistently. It contributes provenance realism and a cascade
hook for the portfolio questions rather than measured difficulty, and its discriminating power at
larger N is untested here.

**Guessing is not a viable path.** The portfolio layer alone requires nine exact answers including
two ID-set questions and a counterintuitive `Redacted` jurisdiction; the risk table requires a
provision-level cite plus an exact category from a closed list of eight, for five contracts; and
the tracker requires 87 of 90 cells correct. The difficulty is distributed rather than
concentrated: 24 distinct tracker cells were missed at least once, only one unanimously, and the
binding constraint differs by layer — redaction-composition rulings in extraction,
provision-selection judgment in the risk table.

*Artifacts: `~/probe_scoring_x7k2/g2_trial_{1..6}_tracker.json`, `g2_analysis.json`,
`g2_analyze.py`; trial directories and transcripts under `~/g2_runs/`.*