# ALE Task Specification — Contract Portfolio Due Diligence (REVISION 1, resubmission)

> **REVISION 1 — response to DeepReview (Minor Revision).** All three recommended changes are addressed: (1) the risk-summary deliverable and its "highest-exposure" criterion are **removed** by author decision, mooting the determinism concern; (2) the shipped grader is repaired (amendment mappings compared as sets; defaults aligned to the shipped scale) and now implements exactly the three layers this spec defines; (3) field 7's tracker validation includes `Redacted`, the tracker Legend is rewritten to the current rules with a single-authority note, and the provenance labels are reconciled to the shipped amendment version. A point-by-point map with verification pointers is in `response_to_review.md`, uploaded alongside this spec. Full disclosure: Deltas item 5 below.

## Task Goal
Perform M&A-style contract due diligence on a difficulty-dense portfolio of 5 commercial agreements and 3 amendments/side letters. Abstract each contract's operative terms into a structured diligence tracker (18 defined fields under a rigorously specified extraction schema), identify which amendment modifies which base agreement (filenames anonymized; parentage determinable only from recitals, party names, dates, and defined terms), reconcile amended terms, and answer 6 compositional portfolio questions with fixed answers. Every contract carries ≥3 schema stress points; the task cannot be completed by shallow extraction.

## Environment
Ubuntu 22.04; LibreOffice 7.x (Calc + Writer) or Microsoft Excel/Word; Tesseract 5.x (or Adobe Acrobat) for the 2 image-based PDFs. No network access required. **Code-execution capability is required** (xlsx outputs cannot be produced by file-write alone; empirically established — file-write-only agent configurations fail 100% independent of extraction quality). Expected expert runtime: ~4 hours for a licensed attorney (author's measured pace: 35–55 min/contract).

## Input Files (agent-visible bundle — 13 files)
- `/contracts/C01.pdf`–`C05.pdf` — 5 executed agreements from SEC EDGAR material-contract exhibits (public filings; full provenance in the evaluator manifest). C01 and C04 are image-based (rasterized from the same public exhibits; OCR pre-verified: ≥98% character fidelity; redaction-marker site-level detectability C01 100%, C04 95.5%, with visual-legibility attestation and the agent-facing C8 artifact instruction).
- `/amendments/Amendment_01.pdf`–`Amendment_03.pdf` — anonymized filenames, stripped metadata. Two genuine EDGAR-filed amendments and one author-drafted amendment (a Second Amendment introducing an aggregate liability cap, drafted against the genuine base agreement and chain-consistent with the genuine First Amendment); authorship flagged per file in the evaluator manifest.
- `diligence_instructions.docx` (v6.8) — 9 global conventions (C0–C8) and 18 field definitions with closed value sets, normalization tokens, and multi-clause composition rules.
- `portfolio_questions.docx` — 6 questions; answers as 9 sub-row IDs.
- `tracker_template.xlsx`, `portfolio_answers.xlsx` — validated templates (dropdown validation, text-formatted columns, canonical-label legend aligned to the v6.8 rules).
- `company_reference.csv` — the "Company" party per contract (perspective anchor for directional fields).

## Required Outputs
1. `diligence_tracker.xlsx` — 90 cells (18 × 5) of operative (post-amendment) terms + the Amended By mapping column.
2. `portfolio_answers.xlsx` — 9 sub-row answers to the 6 questions.

## Evaluation (deterministic; all artifacts included with submission)
Scored by the included `score_task.py` + `scoring_config_N5.json` against reference outputs produced personally by the task author (licensed attorney):
- **L1 — Amendment mapping**: Amended By column, exact match (multi-value per convention C6).
- **L2 — Field extraction**: 90 cells vs. the frozen answer key; pass ≥96% (≤3 wrong cells); multi-value cells order-insensitive; closed sets and normalization tokens throughout.
- **L3 — Portfolio questions**: all 9 sub-rows exact; ID-set sub-rows (Q2b, Q5) compared as sets. Questions are compositional (subset-gate × extracted value); each amendment is load-bearing (the drafted amendment flips two answers; the genuine amendments gate two questions' subset identification); guessing analysis included (joint blind-guess probability ≪0.1%).
- **Overall pass = L1 ∧ L2 ∧ L3.** (L1 multi-value mapping cells and L3 ID-set answers are compared as sets by the shipped harness.)

## Reference Outputs & Authorship
`answer_key_tracker_FROZEN.xlsx`, `portfolio_key.xlsx` — produced by the task author personally performing the full workflow; frozen after a documented audit protocol (below). `answer_key_notes.md` ships alongside: pin-cited rationales for every judgment cell, all adjudication rulings, provenance sign-offs, and the freeze record. Author: [Shuang Liu], licensed attorney (New York State Bar). Contact: [shuangl4@andrew.cmu.edu].

## Key-Quality Protocol (methodology disclosure)
The answer key was validated through: (1) the author's blind test-drive of 4 of 5 contracts (3 composition-rule cells required a corrected second pass — early evidence the schema defeats fast reading even by its author); (2) a structural linter (`key_lint.py`, included) enforcing value-set membership, N/A-logic dependencies, formats, and cross-field consistency; (3) two multi-agent consensus audits (6-trial probe; 4-trial G1) whose consensus-against-key flags were individually adjudicated by the author — several key cells and two amendment-drafting defects were corrected as a result, with every ruling recorded; (4) a post-freeze change bar (demonstrated defect only) under which one instruction gap (v6.6's field-10 proviso sentence) and two evaluator-side scoring rules were fixed after the final evidentiary run, with no key cell edited post-freeze.

## Difficulty Evidence (two configurations, full task, final materials)
**G2 — reference second-tier configuration (claude-sonnet-4-6, default effort, n=6):**
- **Overall pass: 0/6** (unchanged under the Revision 1 layer set: L1 ∧ L2 ∧ L3). Layer aggregates: L1 6/6; L2 0/6 (accuracy 86.7–92.2%, mean 89.3%, vs. the 97.8% bar); portfolio 2/6.
- L2 is the universal blocker.
- Validated published-rule discriminators: the C8-priority composition rule (1/6) and the cure-period required-condition rule (4/6) — both legally correct, both rules published in the instructions, both defeating calendar/pattern intuition. A third historical discriminator (field-10 proviso reading, 0/6) is deliberately **withdrawn** as a difficulty claim: its miss rate reflected an instruction gap the v6.6 sentence closed.
- Honest caveats: L1's 6/6 does not validate amendment mapping as a difficulty source at this N (3 amendments × 5 parents is small); the evidence was measured on v6.5 instructions and the v6.6/v6.7 changes are solvability-increasing, so 0/6 is an upper bound on difficulty-as-shipped; measurements are on the reference second-tier agent per the benchmark's classification guidance.
- Author blind test-drive timings: 35–55 min/contract; full expert runtime ~4 hours.

**G3 — top-tier configuration (claude-opus-4-8, --effort max, verified active per transcript; n=4):**
- **Overall pass: 0/4** (unchanged under the Revision 1 layer set). L1 4/4; L2 0/4 (accuracy 93.3–94.4%, mean 93.9%, best trial two cells short of the bar); portfolio 1/4.
- The configurations fail differently: G2 blocked on extraction breadth (mean L2 89.3%, 24 distinct missed cells), G3 on composed-clause judgment within extraction (accuracy improved to 93.9% with variance collapsed and 9 distinct missed cells — yet the surviving misses concentrate on composition-rule cells, and the portfolio layer dipped). Exemplar: all four G3 trials mis-resolved a composed liability clause by pattern-matching "combined aggregate limit" + "indemnification obligations" without parsing the grammatical object (the capped quantity is the reduction in the indemnity, not the indemnity) — a judgment-layer failure mode distinct from G2's breadth failures.
- Cross-configuration discriminator: the C8-priority composition rule held at 2/10 across both runs; five of six previously adjudicated watch-cells resolved to unanimity at the stronger configuration.
- G3 surfaced one instruction gap (inference of absence from full-provision redaction), closed by a v6.7 clarifying sentence post-run under the same one-way disclosure: measured results are upper bounds on difficulty-as-shipped.
- **Classification note**: the measured 0/6 pass rate sits below the ~50% near-term reference point; six trials cannot statistically resolve how far below (0/6 is consistent with true rates up to ~40%). Failure margins are consistent and substantive but not structural (best trial missed L2 by 4 cells; 2/6 cleared the portfolio layer). Difficulty classification is deferred to the benchmark team's reference panel; the evidence here is offered without a band claim.

## Deltas from the accepted idea-evaluation spec (disclosed)
1. Amendment change-list: "liability-term modification (drafted); waiver-exception and commercial-terms amendments (genuine, field-null)" — the governing-law and term-mechanics targets were dropped when two genuine EDGAR pairs (superior authenticity and parent-matching difficulty, including an embedded-distractor exhibit) were adopted; the cascade property is carried by the drafted amendment and the genuine amendments' mapping-gates.
2. OCR criterion: strict marker survival → site-level detectability (≥95%) + visual attestation + agent-facing artifact instruction; C04 ships at 95.5% (weakest-config 90.9% disclosed; single missed site is a line-wrap artifact, rescue attempted and reverted with measurements recorded).
3. Evaluator-side scoring rules R1 (section-word normalization) and R2 (the C04 acceptance set) adopted after the evidentiary run, both disclosed with measured effects (risk aggregate 14/30 → 19/30; overall pass unchanged at 0/6).
4. Code-execution environment requirement added (empirical finding).
5. **Revision 1 (post-DeepReview):** the per-contract risk-summary deliverable was removed by author design decision, resolving the reviewer's determinism concern (the "highest-exposure" superlative) by removal rather than repair; the reviewer's remaining findings were repaired directly — solver-template/key conflict (field 7 `Redacted` added to the tracker validation), stale tracker Legend rewritten to the current rules with a single-authority note, the shipped grader extended (L1 multi-value mappings compared as sets) with defaults aligned to the shipped scale, and the provenance labels reconciled to the shipped amendment version. The retired risk keys and their clause-cited rationales are preserved in the project archive; the difficulty-run reports' risk sections stand as historical record of the superseded layer.

## Provenance & License
Evaluator manifest: EDGAR accession numbers and URLs for all genuine documents; per-amendment genuine/drafted flag; company-designation chains for the two non-signatory-filer contracts recorded with sign-offs. The drafted amendment is an original work of the task author. All materials CC BY 4.0.

## Packaging (two archives; keys never ship beside the bundle)
- `agent_bundle.zip` — the 13 agent-visible files only; verified by the included `verify_bundle.py` (inventory, metadata, text-layer, leak-scan, and key-blacklist checks) as packaged.
- `evaluator_materials.zip` — frozen keys (tracker/portfolio/risk), `answer_key_notes.md`, `score_task.py` + configs, `key_lint.py`, `verify_bundle.py`, `final_manifest.csv`, `amendment_filename_map.csv`, `guessing_analysis.md`, `ocr_verification.md`, and the source markdown of the instructions and drafted amendment.
- The full difficulty-run reports (`g2_report.md`, `g3_report.md`) — the primary records behind the Difficulty Evidence section — are submitted separately as Difficulty Self-Test evidence on the submission form rather than packaged with the grading materials.