{"claim":{"id":"3fcaf393-e23b-472d-8393-304b5f5f0296","submission_id":"640d9d12-886f-4a4e-8111-145a15943842","submission_revision_id":"eb2a7648-f787-49f3-9c16-337b3aae61ff","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:36:28.638823Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"640d9d12-886f-4a4e-8111-145a15943842","submission_revision_id":"eb2a7648-f787-49f3-9c16-337b3aae61ff","revision_version":3,"submission_status":"under_review","submitted_at":"2026-10-02T16:06:10.269333Z","submitter_name":"Yihao Liang","submitter_email":"yihaoliang.dev@gmail.com","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5.5","harness":"Claude Code subagent with Python/ObsPy (its own detector, picker, associator and locator). There was no GUI and no network, and it saw only input/. One run, 29 min. Result 0.0: detection 19/19 and ML within 0.3 for 94.7%, but only 16/19 events within 3 km (17 needed) and P picks at 63% (70% needed).","estimated_score":"10–20%"}},"submitted_at":"2026-10-02T16:06:10.269333+00:00","industry_domain":"Seismology / Seismic Network Operations (Earth Sciences)","task_short_name":"seismic_catalog_imperial_valley_1","operating_system":"Linux","task_description":"Objective: Work as a seismic network analyst and rebuild the analyst-reviewed earthquake catalog for 30 minutes of a real earthquake swarm (2024-02-12, El Centro, Imperial Valley, California) from raw continuous waveforms. The agent must detect every earthquake with ML >= 1.5 (19 events, ML 1.5-4.3; several are seconds apart or inside the coda of larger events), pick P and S arrivals, associate them into events, locate each event with the network's 1D model and station delays, and compute ML exactly as the network defines it. Regional network analysts (e.g. at Caltech/USGS) do this daily, and the graded reference is their published reviewed catalog. An experienced analyst needs most of a working day.\n\nStarting State: input/waveforms/*.mseed holds 08:58-09:30 UTC continuous data from 30 CI stations (27 three-component broadband, 3 vertical short-period). The other inputs are: input/stations.xml (full responses), input/velocity_model.txt (Hadley-Kanamori 1D, Vp/Vs 1.73), input/station_delays.csv (P/S corrections), input/ml_method.txt (the ML definition: Wood-Anderson, Hutton & Boore 1987; the window ends 0.5 s before the next event's P arrival) and input/ml_corrections.csv (per-channel corrections). DATA_LICENSE.md, in the input zip, carries the SCEDC licence grant and the required attribution. SeisComP 6, Pyrocko Snuffler, NonLinLoc 7 and ObsPy 1.5 are installed. The network is off (mode \"off\") and no catalog is available.\n\nInstructions & Rules:\n1. Detect all earthquakes in 08:59:00-09:29:00 UTC.\n2. Pick P and S arrivals and associate them into events.\n3. Locate with the provided model and station delays.\n4. Compute ML per ml_method.txt with ml_corrections.csv.\n5. Save QuakeML 1.2 to output/catalog.xml. Each event needs an Origin (time, lat, lon AND depth in m), a Magnitude of type \"ML\", and the P/S Picks used: at most one P and one S pick per station (network \"CI\", station, phase_hint, time).\n6. Work only from the provided data.\n\nRevision 2 (2026-10-02), after DeepReview: added DATA_LICENSE.md (SCEDC's published licence grant and attribution; a request for written confirmation will be sent by the contributor); G0 now fails a missing depth or a non-QuakeML-1.2 file (14 tests); ml_method.txt now states the exact window truncation used in calibration. Revision 3 (2026-10-02), after the second DeepReview: at most one primary pick per (network, station, phase) per event is enforced (duplicates fail G4), and picks are matched on the full network identity. The attached zip contains the reference, the grader (17 tests), an inputs-only achievability baseline, the build tools, a draft task_card.json, VERIFICATION.md and the blind self-test.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Run: python grader/grade.py --reference reference/reference_catalog.xml --submission output/catalog.xml (Python 3 + ObsPy, a few seconds). Scoring is binary: 1.0 only if all gates pass.\n\nG0: the file is QuakeML 1.2 (namespace http://quakeml.org/xmlns/bed/1.2) and parses. Every event's origin has time, latitude, longitude AND depth (a missing depth fails), and every event has an ML.\n\nG1 detection: >= 90% of the 19 reference events with ML >= 1.5 are matched one-to-one (|dt| <= 2 s, epicenter <= 5 km). At most 2 submitted events with ML >= 1.8 may match no reference event of any magnitude. The window holds 44 reviewed events; the 25 below ML 1.5 are not required and serve only to decide whether an extra submitted event counts as a false detection.\n\nG2 location: epicenter median <= 1.5 km and >= 85% within 3 km; depth median |dz| <= 2.5 km and >= 80% within 4 km; >= 80% with origin time within 0.6 s.\n\nG3 magnitude: >= 85% within 0.3 ML units, and none worse than 0.6.\n\nG4 picks: each submitted event may carry at most one pick (its primary pick) per (network, station, phase); any duplicate fails G4, so dense guess grids are rejected. Picks are matched on the full identity (network \"CI\", station, phase): >= 70% of the reference manual P picks within 0.15 s, and >= 50% of S picks within 0.30 s.\n\nReference: USGS ComCat analyst-reviewed Hypoinverse origins, ML and manual picks (public domain), restricted to the 30 task stations (1,029 picks). The station delays and ML corrections were calibrated on 105 reviewed events outside the window, using exactly the window rule stated in ml_method.txt. An inputs-only pipeline with perfect picks passes every gate (epicenter median 0.64 km, 94.7% within 3 km, ML within 0.3 for 94.7%); see VERIFICATION.md. 17 grader tests, including a missing depth, a non-QuakeML-1.2 file, a quarter-second guess grid, a single duplicate pick and wrong-network picks.\n\nData licensing: SCEDC's published grant (AWS Registry of Open Data) covers \"use, reproduce and publicly display in all media public data from the Southern California Seismic Network\", with citation (doi:10.7909/C3WD3xH1, doi:10.7914/SN/CI). The exact text and sources are in DATA_LICENSE.md. The contributor will also ask SCEDC for written confirmation and attach the reply when it arrives.","software_and_version":"SeisComP 6 (scolv, scrttv); Pyrocko Snuffler; NonLinLoc 7; ObsPy 1.5","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"10f94c15-b676-427d-8728-3218d5e6d043","filename":"seismic_catalog_input.zip","category":"input_materials","content_type":"application/zip","size_bytes":19060070},{"file_id":"1a386278-7292-4ec1-95c5-c94f83318104","filename":"seismic_catalog_reference_and_grader.zip","category":"reference_output","content_type":"application/zip","size_bytes":550060}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — Revision 3 provides a complete, professionally representative task with usable inputs, corroborated reference evidence, and scoring safeguards consistent with the stated requirements.\n\n**Completeness Check:**\n\n- Instruction: Complete — The analysis window, required workflow, magnitude method, pick constraints, and QuakeML output requirements are explicit.\n- Inputs: Complete — All 30 waveform files decode successfully, providing 84 continuous streams without detected gaps; responses, 60 station-phase delays, and 54 horizontal-channel magnitude corrections cover the supplied data.\n- Toolchain: Complete — The declared Linux tools provide a viable offline, single-machine workflow; their actual installation was not verified.\n- Reference output: Complete — The catalog contains 44 events, including 19 required events, and 1,029 manual picks. Origins, magnitudes, and picks agree with the supplied ComCat source records.\n- Rubric: Complete — The grader implements the stated gates, including duplicate-pick rejection and full network identity matching. All 17 test definitions were inspected.\n\n**File and Task Reachability:**\n\nAttachments: `reachable`. Task: `reachable`.\n\nBoth `seismic_catalog_input.zip` and `seismic_catalog_reference_and_grader.zip` were fully inspected. Their byte counts match the manifest, archive integrity checks pass, and no encryption, unsafe paths, or unsafe member types were found.\n\nInspection covered all waveform members; `stations.xml`, both correction tables, `velocity_model.txt`, `ml_method.txt`, and `DATA_LICENSE.md`; the reference and baseline catalogs, event table, all 44 source XML files, grader and tests, build scripts, task card, verification documents, and self-test artifacts. No attachment was left unreviewed.\n\nThe completion path is offline waveform analysis, arrival picking and association, model-based location, corrected ML calculation, and QuakeML evaluation. Independent catalog comparisons reproduce the baseline’s passing metrics and the self-test’s location and P-pick failures. Submitted programs were not executed; full schema validation, installed-toolchain behavior, and external licensing sources were not independently verified.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\n`public_benchmark` — The protected answer is explicitly identified as a published ComCat catalog for a precise earthquake window. Offline execution and withholding all reference, baseline, self-test, and evaluator assets mitigate answer retrieval. Current public availability was not independently checked; canonical benchmark memorization is not established.\n\n**Recommended Changes:**\n\nThe revision addresses the prior depth, namespace, window-truncation, and duplicate-pick concerns, with supporting evidence. No contributor changes are required. Preserve the offline environment and solver/evaluator separation when deploying the task.","reviewed_at":"2026-10-02T16:19:56.179837Z"},"claim":{"id":"3fcaf393-e23b-472d-8393-304b5f5f0296","submission_id":"640d9d12-886f-4a4e-8111-145a15943842","submission_revision_id":"eb2a7648-f787-49f3-9c16-337b3aae61ff","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:36:28.638823Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}