{"submission_id":"76227f5f-51c6-4590-a6b8-70ba8d217a8f","submission_revision_id":"152321f1-2777-4f85-8550-6dec9715a1be","revision_version":2,"submission_status":"submitted","submitted_at":"2026-07-24T17:15:11.330112Z","submitter_name":"Himanshu Gupta","submitter_email":"him141195@gmail.com","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"Claude Sonnet 4.6","harness":"Claude Code","estimated_score":"<1%"}},"submitted_at":"2026-07-24T17:15:11.330112+00:00","industry_domain":"Software Engineering / Formal Verification (formal methods, interactive theorem proving)","task_short_name":"imo2005p6_lean_v1","operating_system":"Linux","task_description":"Produce a complete, machine-checked Lean 4 (Mathlib) proof of a given theorem  a formal-verification workflow. The theorem is a formalization of IMO 2005 Problem 6 (a combinatorial extremal result), but the graded skill is formal-methods\n  engineering: turning a fixed specification into a kernel-checked proof artifact, not \"solving a contest problem.\"\n\n  Starting State: A single file `input/Problem.lean` containing the exact theorem statement `imo2005_p6` with fixed auxiliary definitions (`solvedCount`, `numSolvedExactly`) and a `sorry` placeholder for the proof. The statement: in a competition with 6 problems where every pair of problems was solved by more than 2/5 of the contestants and nobody solved all 6, at least two contestants each solved exactly 5 problems. It is stated over an arbitrary finite contestant type `C` with a decidable relation `solved : C → Fin 6 → Prop`. Lean 4.27.0 and the prebuilt Mathlib v4.27 cache are installed at setup (paths in input/LEANBIN and input/LEANPATH).\n\n  Instructions & Rules: Create `output/solution.lean` — the same file with the `sorry` in `imo2005_p6` replaced by a complete proof. Keep the `imo2005_p6` signature and the two auxiliary definitions exactly as given; auxiliary lemmas may be added before the theorem. The proof must contain no `sorry`, no `admit`, no `native_decide`, and no newly declared `axiom`, and must compile against\n  Mathlib.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"  Deterministic, layered gate (all must pass for score 1.0, else 0.0):\n\n  1. SIGNATURE PIN: output/solution.lean must contain the imo2005_p6 declaration with byte-for-byte the given signature (name, hypotheses, conclusion) and the exact solvedCount / numSolvedExactly definitions. Rejects weakened or vacuous restatements.\n\n  2. SOURCE SCAN: reject if the source contains `sorry`, `admit`, `native_decide`, or a new `axiom`.\n\n  3. COMPILE: `lean output/solution.lean` (with LEAN_PATH set to the Mathlib cache) must compile with no `error:`.\n\n  4. AXIOM AUDIT: append `#print axioms IMO2005P6.imo2005_p6`, recompile, and require the reported axiom set to be a SUBSET of {propext, Classical.choice, Quot.sound} — the three standard Mathlib axioms. This rejects `sorryAx` (holes), any declared axiom, and `Lean.ofReduceBool` (from native_decide).\n\n  Score = 1.0 iff all four pass, else 0.0. Because the theorem is universally quantified over an arbitrary finite type, it cannot be closed by `decide`/`native_decide`; only a genuine proof passes. A complete grader implementation, an adversarial test battery, and the reference proof are provided in PR #56: https://github.com/rdi-berkeley/agents-last-exam/pull/56","software_and_version":"Lean 4 (v4.27.0) with Mathlib (v4.27)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"c1a76a3d-36e8-43ae-a19f-193c80207a5f","filename":"Problem.lean","category":"input_materials","content_type":"application/octet-stream","size_bytes":2600},{"file_id":"d8d9e470-dc62-4e0c-a1e1-3478038681a3","filename":"reference_solution.lean","category":"reference_output","content_type":"application/octet-stream","size_bytes":37670},{"file_id":"6cce1a81-0e86-4d78-81ac-990b8a8725d0","filename":"evidence.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":582}],"deep_review":{"overall_verdict":"Borderline Accept","review_text":"**Overall Verdict:** Borderline Accept — The task is complete and technically reachable, but its public identification of the exact reference proof and grader requires a small isolation and leakage clarification.\n\n**Completeness Check:**\n- Instruction: Complete — The required artifact, immutable definitions and signature, prohibited constructs, and output location are explicit.\n- Inputs: Complete — `Problem.lean` contains the stated definitions, theorem, and intended proof hole.\n- Toolchain: Complete — Lean 4.27.0, Mathlib v4.27, Linux, and the expected invocation are specified using legally obtainable open-source software.\n- Reference output: Complete — `reference_solution.lean` is a substantial self-contained proof preserving the pinned declarations; static inspection found no proof holes, newly declared axioms, or prohibited tactics.\n- Rubric: Complete — Signature pinning, source scanning, compilation, axiom auditing, and binary scoring are deterministic and objective.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. The manifest, `Problem.lean` (2,600 bytes), `reference_solution.lean` (37,670 bytes), and `evidence.txt` (582 bytes) were fully inspected. All attachments are readable UTF-8 text whose byte counts match the manifest; there are no containers, encryption barriers, executables, or unreviewed contents. The Lean files depend only on Mathlib, and the reference preserves the exact definition bodies and theorem declaration. The Lean sources were not executed under this evidence-safe review, so compilation and axiom output were not independently reproduced; this is a reviewer safety limitation rather than a submission defect.\n\nTask classification: `reachable`. A solver can open `Problem.lean`, replace the proof hole while preserving the specification, write `output/solution.lean`, and be evaluated through the signature, source, compile, and axiom-audit gates on one ordinary machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The task explicitly identifies canonical IMO 2005 Problem 6 and states that a public development artifact contains the exact reference proof and grader. A fully offline solver environment can mitigate direct retrieval, but that isolation is not established in the submitted evidence; familiarity with the canonical mathematics is less concerning than access to the bespoke Lean proof.\n\n**Recommended Changes:**\nThe formal specification, reference proof, and deterministic evaluation design are strong. Remove the protected-reference/grader location from solver-visible material, keep reference assets evaluator-only, and explicitly document or enforce a network-disabled solver environment.","reviewed_at":"2026-08-12T10:21:17.631321Z"},"claim":null}