{"claim":{"id":"b01f92de-0030-474f-b858-042810048d38","submission_id":"640c0198-2e2c-430c-b560-c29ce123de32","submission_revision_id":"bcb946cb-67c8-43b3-b731-56d7a5eea041","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:41.224121Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"640c0198-2e2c-430c-b560-c29ce123de32","submission_revision_id":"bcb946cb-67c8-43b3-b731-56d7a5eea041","revision_version":2,"submission_status":"under_review","submitted_at":"2026-07-23T21:57:05.548132Z","submitter_name":"Dongrong Yang","submitter_email":"hlthrevolution21@gmail.com","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"Claude Sonnet 5","harness":"claude code","estimated_score":"<1%"}},"submitted_at":"2026-07-23T21:57:05.548132+00:00","industry_domain":"Healthcare (Radiation Oncology)","task_short_name":"\ttg119_imrt_pyradplan_planning","operating_system":"Linux","task_description":"Objective: Produce an IMRT beamlet-intensity treatment plan for the AAPM\nTG-119 C-shape commissioning phantom that meets four dosimetric\nrequirements simultaneously — including one requirement that cannot be\nmet by parameter tuning alone.\nStarting State: The TG-119 phantom (CT + structure set: OuterTarget,\nCore, BODY) ships inside the pyRadPlan library itself, loaded via\npyRadPlan.load_tg119(). The task provides a prescription/acceptance-\ncriteria file (prescription.json), a full agent workflow README\n(README_AGENT.md), and a one-line pip installer (setup_pyradplan.sh).\nInstructions & Rules: Choose a coplanar beam arrangement (≤9 beams,\nbixel width ≥5mm) and tune the structure set's optimization objectives\nso the plan achieves, simultaneously: OuterTarget D95≥50Gy, OuterTarget\nD10≤55Gy, Core D5≤10Gy (published AAPM TG-119 criteria), and a\ndose-falloff/gradient requirement (gradient_ratio≤3.80). The gradient\nrequirement is calibrated so it cannot be reached by retuning the three\nbuilt-in structures alone — the agent must recognize this and construct\nits own additional optimization structure to reach it. Do not modify\nfiles under input/; write only beam_geometry.json, weights.npy,\nplan_metrics.json, and report.md under output/.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Before any dose calculation runs: beam_geometry.json must specify 1–9 coplanar beams (all couch angles = 0) with bixel width ≥5mm, and weights.npy must be a non-negative array whose length exactly matches the beamlet count implied by that geometry. Any violation — wrong weight-vector length, negative weights, >9 beams, non-coplanar angles — scores 0.0 immediately, before any dose is computed.\n\nThe evaluator reconstructs dose from the submitted beam geometry and weights, then computes 4 hard requirements: OuterTarget D95≥50Gy, D10≤55Gy, Core D5≤10Gy, and gradient_ratio≤3.80 (a dose-falloff/spillage measure — the volume of BODY receiving ≥50% of the prescription dose divided by target volume — that cannot be met by retuning the phantom's shipped structures alone; it requires the agent to construct and tune a new, self-designed optimization structure). All 4 must be met within tight tolerance (0.5 Gy on the three dose-based metrics, 0.02 on gradient_ratio — solver/discretization noise allowances, not leniency bands).\n\nIf, and only if, all 4 hard requirements are met, additional partial credit is added for 3 bonus quality margins: Core sparing below the 10Gy limit, target D10 homogeneity below the 55Gy limit, and plan conformity (how little prescription-level dose spills outside the target). Bonus never substitutes for a missed hard requirement — a submission failing any of the 4 core criteria scores 0.0 regardless of bonus performance.\n\nThe final combined score averages all 4 core criteria plus the 3 bonus criteria (7 terms total, bounded in [0,1]). A submission counts as \"success\" only when hard_gates_ok is true AND the combined score reaches ≥0.65 — this threshold is calibrated against my own verified best-known solution (a 9-beam plan using a purpose-built annular \"Ring\" structure around the target), which scores 0.712. 0.65 leaves headroom above a bare hard-gate pass (which alone, with zero bonus margin, scores ~0.565) while remaining reachable by the documented winning configuration, so the bar requires genuine quality, not just technical compliance.","software_and_version":"pyRadPlan 0.4.1 (Python, pip-installable)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"a81943a8-efb5-448e-a4da-c35dc0ce8cb0","filename":"input_submit.zip","category":"input_materials","content_type":"application/zip","size_bytes":9227},{"file_id":"e1329c45-e494-4ead-a4b8-cea16250c3f8","filename":"tg119_self_test_evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":5139},{"file_id":"a19baa07-31d7-4619-b279-17171e43ad10","filename":"output_submit 2.zip","category":"reference_output","content_type":"application/zip","size_bytes":49134}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, coherent, objectively gradable, and supported by a valid reference plan and internally consistent calibration evidence.\n\n**Completeness Check:**\n- Instruction: Complete — The planning objective, fixed prescription, beam constraints, optimization freedom, required outputs, and dose-metric definitions are explicit.\n- Inputs: Complete — The agent README and valid prescription JSON explain the bundled phantom, structures, constraints, and self-check calculations.\n- Toolchain: Complete — A pinned pyRadPlan 0.4.1 installation path, Linux environment, dependencies, API workflow, and known DVH workaround are documented.\n- Reference output: Complete — All four required artifacts are present. The reference geometry specifies nine coplanar beams and 5 mm bixels; `weights.npy` is a valid NumPy v1 little-endian float64 vector with shape `(2851,)`, containing only finite, non-negative values.\n- Rubric: Complete — The evaluator independently reconstructs dose, verifies geometry and weight length, applies four hard gates with stated tolerances, computes three bonus terms, and enforces hard-gate failure as a final zero.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. All three manifest attachments were downloaded at their declared sizes, identified as valid unencrypted ZIP containers, integrity-tested, and safely inspected. The inspected content included `README_AGENT.md`, `prescription.json`, `setup_pyradplan.sh`, `difficulty_self_test_evidence.md`, `random_search_results.json`, `main.py`, `EVALUATOR_README.md`, `task_card.json`, `score_outputs.py`, and the four reference-output files. All substantive content was reviewed; benign Finder metadata and AppleDouble resource records were inventoried but not semantically decoded. The self-test data reconciles to 24 trials, zero hard-gate passes, the declared seed, and 1,154.4 seconds total. Submitted programs were not executed, and the dose-engine result could not be independently reproduced in the review environment, so runtime claims remain supported by internally consistent submitted evidence rather than reviewer reproduction. The completion path is clear: install the pinned package, load its bundled phantom, optimize beamlet weights, write the four required outputs, then reconstruct dose from pristine geometry and independently score the clinical and quality metrics. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The submission affirmatively identifies the AAPM TG-119 C-shape phantom and three acceptance criteria as published benchmark material. The custom gradient requirement, bonus scoring, and reference beamlet weights are task-specific, which materially reduces direct retrieval of a complete answer. Public availability was not independently checked under the review evidence restrictions.\n\n**Recommended Changes:**\nThe reference artifacts, evaluator, and calibration evidence are unusually thorough and mutually consistent. No material changes are required. Optional evaluator hardening would be to require the declared `plan_metrics.json` and `report.md` files and reject non-finite or even minimally negative weights explicitly, matching the written preflight contract exactly.","reviewed_at":"2026-08-17T22:49:38.967171Z"},"claim":{"id":"b01f92de-0030-474f-b858-042810048d38","submission_id":"640c0198-2e2c-430c-b560-c29ce123de32","submission_revision_id":"bcb946cb-67c8-43b3-b731-56d7a5eea041","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:41.224121Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}