{"claim":{"id":"e3d60a6e-4a8e-41e8-8f03-77e96937d5cc","submission_id":"8f386ec5-869a-4002-b4a4-2ebdb7c217fa","submission_revision_id":"c77f68ed-4f32-4873-be2e-85b6c2e8a6b5","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:33.665416Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"8f386ec5-869a-4002-b4a4-2ebdb7c217fa","submission_revision_id":"c77f68ed-4f32-4873-be2e-85b6c2e8a6b5","revision_version":3,"submission_status":"under_review","submitted_at":"2026-07-21T23:08:50.354677Z","submitter_name":"Kerui Wu","submitter_email":"keruiwu@umass.edu","task":{"status":"submitted","version":3,"extra_fields":{"drive_link":"https://drive.google.com/drive/folders/1BfkYxUFPed83qfI_atMWKvQrMgEq07cF?usp=sharing","difficulty_self_test":{"model":"Claude Sonnet 5","harness":"Claude Code","estimated_score":"20–40%"}},"submitted_at":"2026-07-21T23:08:50.354677+00:00","industry_domain":"Health & Medicine — Clinical Neurophysiology","task_short_name":"eeg_seizure_annotation","operating_system":"Linux","task_description":"Objective: Review one long-term de-identified scalp EEG recording end-to-end and annotate every electrographic seizure it contains, including each event's onset and duration.\n\nStarting State: The agent's working directory contains input/recording.edf (~50 min, 23-channel bipolar longitudinal montage, 256 Hz, header-sanitized), input/task.md (full spec: onset/duration definitions, montage, output contract),\ninput/review_report_template.md, input/empty_events.tsv (header contract), and\ninput/runtime_env/pyproject.toml (pinned analysis runtime). software/ carries\nbootstrap_runtime.sh and python_with_task_deps.sh.\n\nInstructions & Rules:\n1. Bootstrap the task-local venv: bash software/bootstrap_runtime.sh, then run\n   analysis scripts via software/python_with_task_deps.sh.\n2. Load the EDF with pyEDFlib and analyse it with purpose-written code (filtering,\n   spectral and time-domain features). Review the WHOLE recording.\n3. Separate electrographic seizures from muscle, movement, electrode and chewing artifact, and from benign rhythmic variants. \n4. Onset = first moment of unequivocal, sustained electrographic seizure activity, in seconds from EDF start (not the first isolated spike). Duration = seconds until evolving ictal activity ends.\n5. The recording may contain ZERO, one, or several seizures; ~17% contain none, so a false event is penalised.\n6. Write output/events.tsv (tab-separated, first three columns exactly onset, duration, trial_type; one row per seizure, sorted; trial_type literally \"seizure\"; header-only if none) and output/review_report.md filled from the template and consistent with the TSV.\n7. Prohibited: any external seizure database, published annotation, online lookup\n   of the recording, or a pretrained seizure detector.\n\nTask variants: This task ships as 24 distinct recordings: `base` plus 23 additional\nvariants, all sharing the SAME workflow, scorer, and 100-point rubric, but only the recording and its reference annotation differ per variant. `base` is the published\nalias of instance `eeg_case_df090a`, so of the 24 recordings, 23 are submitted as\ntask variants. The set spans four difficulty classes: hard_negative (4, true-negative\nrecordings with no seizure), single_seizure (5), multi_seizure (8), and short_event\n(7). A reviewer can build all 24 deterministically from the public source with dataset_preparation/prepare_dataset.py (see the package README).","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"THIS IS THE PARENT TASK = eeg_case_df090a (multi_seizure), published as `base`.\nScore against: --reference data/health_medicine/eeg_seizure_annotation_review/base/reference\n\n(base is the alias of df090a so it is byte-identical to .../eeg_case_df090a/reference). The 23 task variants are the other 23 distinct recordings; df090a is NOT re-submitted as a variant. Self-test score for this recording: 21.12 / 100.\n\nSCORING IS PER-INSTANCE: \nEach variant (`base` and every `eeg_case_*`) is graded independently by the same scorer against ITS OWN reference/ directory — same rubric for all, only the --reference path changes. There is NO cross-variant aggregation in grading; the 100-point rubric below applies to one variant at a time. (aggregate_results.py exists only for reporting a sweep's mean/pass-rate, and plays no part in the ALE score.)\n\nHOW TO RUN:\n1. Build the dataset first — no recordings ship:\n```\npip install -r dataset_preparation/requirements.txt\npython3 dataset_preparation/prepare_dataset.py --out data\n```\n(~1.6 GB, deterministic; reproduces `base` = eeg_case_df090a byte-for-byte and adds the other 23 variants under data/health_medicine/eeg_seizure_annotation_review/.)\n\n2. Smoke-test the scorer end-to-end on ONE variant, using the expert answer as a stand-in agent (expect 1.0000):\n```\npython3 baseline/run_baseline.py --oracle --score --variant base\n```\n\n3. Score one agent output against that instance's reference:\n```\npython3 evaluation/scripts/score_outputs.py \\\n         --output    <the agent's output dir for that variant> \\\n         --reference data/health_medicine/eeg_seizure_annotation_review/<variant>/reference\n```\nRepeat step 3 per variant with the matching --reference path. Each call returns\n   that instance's [0.0, 1.0] score independently.\n\n\nVERIFY SELF-TEST:\nConfig: claude-sonnet-5 driven by Claude Code as the agent under test, one run per instance over all 24 distinct recordings, per-solver isolation, no network at solve time (wheels are fetched once into the shared uv cache on the first bootstrap). \n\nPrerequisites on the host:\n- `uv` on PATH (the solve-time bootstrap_runtime.sh builds the pinned 3.11 analysis venv with it; it aborts if uv is missing). Install: `curl -LsSf https://astral.sh/uv/install.sh | sh`\n- `claude` CLI (Claude Code) as the agent harness. \n\nReproduce:\n1. Build the dataset (see HOW TO RUN step 1).\n2. Run the sweep — each (variant, run) gets its own workdir and TMPDIR, so\n   --concurrency is safe:\n```\npython3 baseline/run_baseline.py \\\n       --agent-cmd 'claude -p \"$(cat {prompt})\" --model claude-sonnet-5 --dangerously-skip-permissions' \\\n       --repeats 1 --concurrency 4 --score \\\n       --results results/sonnet5.json --label claude-sonnet-5\n```\n(--dangerously-skip-permissions lets the headless agent run bash / write files\n   unattended; without it the agent refuses every tool and produces nothing.)\n3. Aggregate for the report:\n```\npython3 baseline/aggregate_results.py results/sonnet5.json\n```\nNote on counting: `base` is the published alias of `eeg_case_df090a` (same recording). A raw sweep therefore scores that recording twice. The reported numbers are over the 24 DISTINCT recordings (base alias dropped, df090a kept); the raw per-run results.json and two agent transcripts are attached as Evidence.\n\nSCORING RUBRICS:\n\nHARD GATES — any one triggers score 0.0:\n- events.tsv or review_report.md missing/unreadable\n- events.tsv not tab-delimited, or first three columns not exactly onset, duration, trial_type\n- any onset/duration non-finite, any onset negative, any duration non-positive\n- any event extending beyond recording duration\n- more than 10 false positives per hour\n\nMATCHING: predicted intervals matched one-to-one to reference by maximum-total-temporal-IoU assignment; pairs below IoU 0.25 unmatchable. Solved exactly by pruned DFS (no scipy), deterministic, ties by prediction index.\n\nPOINTS (100 total):\n- Onset accuracy      50 — linear, 1.0 at 0 s error -> 0.0 at 5 s\n- Duration accuracy   35 — linear, 1.0 at 0 s error -> 0.0 at 20 s\n- False-positive ctrl 10 — linear, 1.0 at 0 FP/h -> 0.0 at 2 FP/h\n- TSV schema           3 — all-or-nothing; forfeited on duplicate/overlapping/unsorted rows or wrong trial_type\n- Report consistency   2 — fraction of 6 mechanical checks (status complete, count matches TSV, event-section count matches, per-event onsets consistent, per-event durations consistent, no unfilled placeholders)\n\n","software_and_version":"Python 3.11+, pyEDFlib 0.1.42, numpy 2.4.6, scipy 1.17.1","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"8c98c44d-c644-41cd-aaeb-71ece081bae3","filename":"eeg_evidence_sonnet5.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":7553}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The revision presents a complete, professionally representative EEG annotation task with a reproducible workflow, objective per-instance evaluation, and credible difficulty evidence.\n\n**Completeness Check:**\n- Instruction: Complete — defines whole-recording review, seizure criteria, onset and duration conventions, artifact exclusions, and exact deliverables.\n- Inputs: Complete — specifies deterministic generation of 24 de-identified EDF instances and the solver-visible specification, templates, TSV contract, and runtime files.\n- Toolchain: Complete — provides compatible Linux and Python requirements, pinned analysis dependencies, bootstrap wrappers, and host prerequisites.\n- Reference output: Complete — identifies protected per-instance references, establishes the base alias relationship, and documents a successful oracle smoke test.\n- Rubric: Complete — includes explicit hard gates, deterministic interval matching, point weights, tolerances, schema checks, and report-consistency checks.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. The inspected attachment was `eeg_evidence_sonnet5.zip`, a valid 7,553-byte, unencrypted ZIP containing six readable ASCII or UTF-8 text/JSON files: `NOTES.txt`, `aggregate_summary.txt`, `sonnet5.json`, `sonnet5_24distinct.json`, `eeg_case_e0755c_score0.98_stdout.log`, and `base_score0.37_stdout.log`. There were no unsafe paths, links, devices, encrypted members, executable payloads, integrity errors, or unreviewed archive content. The JSON schemas and all 24 distinct result records were inspected; their counts, class summaries, 21.125/100 parent-instance score, aggregate statistics, and alias handling are internally consistent. The generated EDF corpus, preparation source, templates, references, and scorer were not manifest attachments, so their bytes could not be independently inspected; external provenance checks were also outside the permitted evidence boundary. The declared path is nevertheless complete: build the dataset, bootstrap the pinned runtime, analyze each EDF offline, produce the two contracted outputs, and score each variant against its corresponding evaluator-only reference. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n- `public_benchmark` — the recordings are deterministically derived from a public source, and the task anticipates possible published-annotation lookup. Header sanitization, opaque instance names, prohibition of external lookup, and no-network solver isolation mitigate direct answer retrieval. External search was not permitted, so canonical-benchmark or pretraining exposure could not be independently established.\n\n**Recommended Changes:**\nThe submission provides a precise clinical workflow, strong evaluator specification, and internally consistent self-test evidence. No changes are required.","reviewed_at":"2026-08-15T00:05:16.010474Z"},"claim":{"id":"e3d60a6e-4a8e-41e8-8f03-77e96937d5cc","submission_id":"8f386ec5-869a-4002-b4a4-2ebdb7c217fa","submission_revision_id":"c77f68ed-4f32-4873-be2e-85b6c2e8a6b5","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:33.665416Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}