{"claim":{"id":"16ddb4a4-137e-4c90-969f-11c9429417a4","submission_id":"159364fb-373d-41ea-a3de-67fd60db83b6","submission_revision_id":"eb093d81-370b-461d-87c7-15386baccca5","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:09.108219Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"159364fb-373d-41ea-a3de-67fd60db83b6","submission_revision_id":"eb093d81-370b-461d-87c7-15386baccca5","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-25T02:11:43.585762Z","submitter_name":"JICHENG WANG","submitter_email":"jconliner67@gmail.com","task":{"status":"submitted","version":3,"extra_fields":{},"submitted_at":"2026-08-25T02:11:43.585762+00:00","industry_domain":"Software Engineering / AI Benchmark Quality Assurance","task_short_name":"contractbench_oracle_sweep_1","operating_system":"Linux","task_description":"Objective: Audit a public agent benchmark by running every task's own reference (\"oracle\") solution against that task's own verifier at a pinned revision, and produce a machine-readable report of what the verifiers actually returned. This is routine benchmark-maintenance work: before anyone trusts a suite's scores, someone has to confirm the suite's reference solutions still earn the rewards they are supposed to earn.\n\nStarting State: Nothing is set up. You are given the public repository https://github.com/SecurityLab-UCD/ContractBench and the exact commit to audit: c50eefee49b6925e2ccbf3c51a987ed705148725. (Use this commit or later; earlier revisions contain since-fixed task-server and reference-solution bugs, so an older checkout will not reproduce a clean audit.) The self-contained tasks live under harbor/cookbook-export/, each with environment/Dockerfile, solution/solve.sh, tests/test_outputs.py, and task.toml. Each task pairs a FastAPI task server with a pytest verifier that computes a reward in [0, 1].\n\nInstructions & Rules:\n1. Clone the repository and check out the pinned commit exactly.\n2. Set up whatever tooling is needed to execute the benchmark's Harbor-format tasks. The upstream Harbor CLI (uv tool install harbor, tested with 0.13.x, which ships a free `oracle` agent) is the intended path, but any equivalent runner you build yourself - build image, start server, execute solution/solve.sh, run the tests/ verifier, read the reward - is acceptable.\n3. For EVERY task in the suite: run that task's reference solution against its own task server inside its own Docker environment, then run its verifier, and record the verifier-computed reward. The reward must come from verifier output (Harbor's per-trial result.json -> verifier_result.rewards.reward, or the container's /logs/verifier/reward.txt) - never from agent-side files and never from your own judgement of whether the solution looked correct.\n4. Cross-reference each task's metadata: read `difficulty` and `category` from that task's task.toml, and record the wall-clock seconds its reference solution took.\n5. Emit results.json with exactly this schema:\n\n{\n  \"commit\": \"<the pinned commit you audited>\",\n  \"tasks\": [\n    {\"task\": \"<task directory name>\",\n     \"difficulty\": \"<from task.toml>\",\n     \"category\": \"<from task.toml>\",\n     \"reward\": <float, verifier-computed>,\n     \"passed\": <bool, reward >= 1.0>,\n     \"wall_clock_sec\": <float > 0>}\n  ],\n  \"summary\": {\"n_tasks\": <int>, \"n_passed\": <int>, \"pass_rate\": <float>,\n              \"passed_by_difficulty\": {\"medium\": <int>, \"hard\": <int>,\n                                       \"very_hard\": <int>, \"extreme\": <int>}}\n}\n\nOne entry per task in the suite, sorted ascending by \"task\", with no duplicates and no entries for tasks that are not in the suite. Report what the verifiers returned, whatever that turns out to be - do not assume the suite is clean and do not assume it is broken.\n\n6. PRESERVE AND SUBMIT THE RAW RUNNER OUTPUT. Ship the per-trial result.json files (or equivalent per-task verifier logs) in a trials/ directory next to results.json, so every reward in the report can be traced back to the verifier that produced it.\n\nThe sweep is deterministic (virtual-clock timing, SHA-256 byte checks, no LLM calls, no API keys) and CPU-only. Requirements: a Linux or macOS host with a running Docker daemon, Python 3.10+, and git. Internet access is needed only to clone the repository and pull base images and pip packages.\n\nResource provisioning (measured at concurrency 4 on linux/amd64): a cold run that builds every task image from an empty cache and then executes all trials takes roughly 21 minutes end to end; budget 30. Disk is small because the suite compiles to only two distinct Dockerfiles, so the images share nearly all their layers: about 0.33 GB incremental on a host that already has python:3.12-slim, about 0.45 GB of Docker data on an empty VM, or ~1.6 GB including the Python/Harbor toolchain and the repository clone. Peak memory was 113.6 MiB per container and 361 MiB across four concurrent containers, with the runner process tree peaking at 713 MiB RSS. CPU is not the bottleneck; the wall-clock floor is set by the single longest trial, which deliberately sleeps, so concurrency beyond about 8 buys almost nothing. 8 vCPU / 8 GB RAM / 20 GB disk is comfortable; 4 vCPU / 4 GB / 10 GB is the workable minimum.\n\nDeliverables:\n- results.json in the schema above\n- trials/ containing the raw per-trial verifier outputs","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Fully automated and deterministic, by the attached grade.py. Usage: python3 grade.py <results.json> --trials <raw trial dir>. It prints a final line \"SCORE: <0-1>\". No human judgement and no LLM judging.\n\nIMPORTANT - grading begins with a PROVENANCE GATE, because the report alone proves nothing: the repository, its oracle scripts and its task metadata are all public, so a plausible-looking table can be written without running anything. The gate requires the raw per-trial verifier outputs and checks that:\n  (a) every task in the suite has a usable raw verifier output carrying a numeric verifier-computed reward;\n  (b) there are no duplicate raw trials and none for tasks outside the suite;\n  (c) every reward in results.json equals, to within 1e-6, the reward that task's own verifier actually emitted.\nIf any of these fail, the submission scores 0.0 and the reason is printed. A report with no raw outputs, or whose numbers disagree with them, earns nothing regardless of how correct it looks. (Verified: a report submitted without trials scores 0.0; a report whose reward for one task is altered from the verifier's value is caught by name and scores 0.0.)\n\nSchema is also enforced strictly, before scoring: exactly one entry per task in the suite, unique task names, sorted ascending by \"task\", no unknown task names, and every entry an object with a string \"task\". Any violation fails the submission. (Verified: short reports, reversed ordering and invented task names each score 0.0.)\n\nPast the gate the report is scored field by field with partial credit:\n- 0.80 per-task correctness. For each task: reward matching the reference to 1e-6, plus `passed`, `difficulty`, `category`, and a wall_clock_sec that is present, numeric and positive. The wall-clock VALUE is deliberately not compared because it is machine-dependent; only presence and positivity are. Credit is fractional per task and per field, so a partially correct audit scores proportionally.\n- 0.15 summary correctness: n_tasks, n_passed, pass_rate (1e-6) and passed_by_difficulty, each an equal share.\n- 0.05 the audited commit field.\n\nThe expected per-task rewards, suite size and difficulty totals live only in the evaluator-side reference.json in the attached reference bundle; they are deliberately NOT stated in the solver-visible task text, so a solver cannot know the expected outcome without running the audit. That bundle also contains reference_trials/, the raw per-trial verifier outputs from our own reference run, so your team can validate the grader end to end.\n\nWhy the reference is trustworthy: the benchmark is deterministic by construction (virtual-clock timing, SHA-256 byte checks, no LLM calls, no network-dependent assertions). We ran the full sweep independently on macOS/arm64 and twice on linux/x86-64 hosts, obtaining identical rewards and difficulty breakdowns each time.","software_and_version":"Docker 28.x, Python 3.11+, Git, Harbor CLI 0.13.x (agent-benchmark runner), ContractBench (commit c50eefe)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"3ee88262-d3fb-4525-8112-752365c2b5a9","filename":"01_input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":2938},{"file_id":"e5cbd226-237d-4c1e-a749-b552a8ee28c2","filename":"01_reference_output_and_grader.zip","category":"reference_output","content_type":"application/zip","size_bytes":57589}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, objectively gradable, and feasible on a conventional single-machine Docker setup.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, pinned revision, full-suite scope, execution rules, required output schema, provenance requirements, and deliverables are explicit.\n- Inputs: Complete — The solver-visible material identifies the benchmark source, exact revision, task location, and required repository structure without exposing evaluator reference values.\n- Toolchain: Complete — Docker, Python, Git, and Harbor or an equivalent runner are legally obtainable; supported platforms, resource expectations, runtime, and network requirements are documented.\n- Reference output: Complete — The evaluator bundle contains a reference report, evaluator reference, aggregate run record, and 33 per-task verifier records. Task identities, rewards, metadata, summaries, and commit are consistent; all wall-clock values are positive.\n- Rubric: Complete — The deterministic grader implements schema enforcement, a raw-evidence provenance gate, task-level partial credit, summary scoring, commit scoring, numeric tolerances, and a parseable final score.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both manifested ZIP files were inspected at their declared sizes and matched their declared formats. `01_input_materials.zip` contains one readable UTF-8 Markdown instruction file. `01_reference_output_and_grader.zip` contains 72 entries: 35 directories and 37 files, including 36 valid JSON files and one ASCII Python grader. All members were reviewed through archive listings, integrity checks, static parsing, or representative content inspection; no attachment content remains unreviewed. The archives are readable, unencrypted, modestly expanded, and contain no traversal names, links, device entries, or unexpected binary payloads. The uploaded grader was inspected statically and was not executed. The external public repository and package endpoints were not independently accessed within the evidence boundary, although the complete reference run records support successful execution of the declared workflow.\n\nThe completion path is clear: obtain the pinned repository revision, run each of its 33 oracle solutions in its Docker environment, execute each verifier, preserve the raw verifier results, construct the sorted report, and evaluate both artifacts with the supplied deterministic grader.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The submission affirmatively identifies a public benchmark and public oracle solutions. This is intrinsic to the maintenance task rather than an answer shortcut: the required work is executing every oracle against its verifier, and the provenance gate requires matching raw per-task verifier outputs. Current public availability was not independently verified within the evidence boundary.\n\n**Recommended Changes:**\nThe submission provides unusually strong execution evidence, resource guidance, artifact separation, and provenance enforcement. No changes are required.","reviewed_at":"2026-08-25T02:20:51.507713Z"},"claim":{"id":"16ddb4a4-137e-4c90-969f-11c9429417a4","submission_id":"159364fb-373d-41ea-a3de-67fd60db83b6","submission_revision_id":"eb093d81-370b-461d-87c7-15386baccca5","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:09.108219Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}