{"submission_id":"5dc8159c-8fd4-40aa-902c-72fbb214eaf3","submission_revision_id":"185abc73-10a4-4077-b712-83ab8f864a43","revision_version":2,"submission_status":"under_review","submitted_at":"2026-08-20T09:14:11.363950Z","submitter_name":"Xinyu Guo","submitter_email":"xinyu@abaka.ai","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5 (medium effort)","harness":"ustom bash-tool agent loop","estimated_score":"<1%"}},"submitted_at":"2026-08-20T09:14:11.363950+00:00","industry_domain":"Geospatial Analysis / Hydrology (GIS)","task_short_name":"qgis_hydro_flood_exposure","operating_system":"Linux, MacOS","task_description":"Objective: For each of 20 independent terrain instances, run a full hydrological flood-exposure analysis and report the exact numeric results.\n\nStarting State: instances/01 .. 20, each with its own input/ folder containing:\n- dem_west.tif, dem_east.tif — a DEM delivered as two overlapping tiles. The east tile carries a HIDDEN constant vertical-datum offset that must be detected from the overlap region and corrected before mosaicking.\n- pop_density.tif — population density raster (persons per cell), 400x400 @ 25 m, EPSG:32633.\n- outlet.geojson — the watershed outlet point.\nrules.md in each instance gives the three scenario water levels (S10 / S50 / S100, in metres).\n\nInstructions & Rules (apply identically to every instance):\n1. Reconcile the datum offset between the two DEM tiles and mosaic them into one DEM.\n2. Fill sinks (depression filling).\n3. Compute D8 flow direction, then flow accumulation.\n4. Derive the stream network (accumulation >= threshold).\n5. Delineate the watershed of the outlet point.\n6. Partition the watershed into elevation zones.\n7. For each scenario water level: flood = watershed cells whose filled elevation <= water level; compute flooded cell count, flooded area (m2), exposed population (sum of pop_density over flooded cells), and per-hub exposed population for the four hubs.\n\nOutput: write output/<id>/result.json for every instance, with fields: globals (num_sinks_filled, watershed_cell_count, watershed_area_m2, stream_cell_count, max_flow_accumulation), watershed_zones (per-zone cell counts), and scenarios S10/S50/S100 each with flood_cell_count, flood_area_m2, exposed_population, hub_exposed{hubA..hubD}.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Fully deterministic, provided as evaluate.py (run: python evaluate.py output reference).\n\nFor each instance, evaluate.py walks reference/gt_<id>.json. Every leaf is one checkpoint stored as {expected_value, tolerance_pct}. A checkpoint passes iff |agent_value - expected_value| <= expected_value * tolerance_pct. Tolerances: num_sinks_filled 15%; watershed cell/area/stream/accumulation and per-zone counts 1%; flood_cell_count must match EXACTLY (tolerance 0); flood_area_m2 / exposed_population / per-hub exposed 0.1%.\n\nAn instance PASSES if at most 2 of its ~33-35 checkpoints fail. The task PASS requires ALL 20 instances to pass; the grader also prints the per-datapoint pass rate (e.g. 4/20). Ground truth for every instance was generated by two independent implementations and cross-checked to 0 disagreement.\n","software_and_version":"QGIS 3.34 LTR / GDAL 3.x — or any equivalent Python geo stack (rasterio, geopandas, numpy, scipy). Graded on numeric result only; tool-agnostic.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"fd36bfd0-0022-47f4-8835-3507a200e2cc","filename":"self_test_gpt55_medium.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":3588},{"file_id":"2e1c181b-98d1-4c41-9dfe-c00424607d15","filename":"qgis_flood_v3_multi_INPUT.zip","category":"input_materials","content_type":"application/zip","size_bytes":23899417},{"file_id":"0082afdd-7787-4ef0-bc55-c28a4daa3c72","filename":"qgis_flood_v3_multi_REFERENCE.zip","category":"reference_output","content_type":"application/zip","size_bytes":15682}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, deterministic, safely reachable, independently reproducible, and supported by comprehensive objective evaluation.\n\n**Completeness Check:**\n- Instruction: Complete — Specifies datum reconciliation, priority-flood sink filling, D8 ordering, accumulation, watershed delineation, zoning, scenarios, hubs, and exact output schema.\n- Inputs: Complete — All 20 instances contain distinct, valid DEM tiles, population rasters, outlet GeoJSON, and scenario rules with consistent grids and coordinates.\n- Toolchain: Complete — QGIS/GDAL and equivalent open-source Python stacks provide a legal, practical implementation path on a standard machine.\n- Reference output: Complete — Contains 20 distinct, parseable ground-truth JSON files and an evaluator identical to the solver-visible evaluator.\n- Rubric: Complete — Defines checkpoint tolerances, per-instance failure allowance, all-instance pass criteria, and objective numeric coverage.\n\n**File and Task Reachability:**\nAttachment set: reachable. Task: reachable. The inspected files were `self_test_gpt55_medium.txt`, `qgis_flood_v3_multi_INPUT.zip`, and `qgis_flood_v3_multi_REFERENCE.zip`. Their types and byte counts matched the manifest. Both ZIPs were valid, unencrypted, modestly sized containers without unsafe paths or special entries. The input contained 20 complete instance sets: 60 valid float32 GeoTIFF rasters, 20 valid outlet GeoJSON files, and 20 scenario-rule files. Raster metadata confirmed EPSG:32633, 25-meter cells, expected dimensions and alignment, finite values, and the constant approximately 2.6-meter overlap offset. The reference contained all 20 expected result trees, consistent tolerances, and valid cross-field invariants. No attachment content remained unreviewed, and no material tooling limitation remained. Read-only independent calculation exactly reproduced the principal global values and all 60 exact flood-cell counts, while all 60 total exposed-population results met tolerance. Solver-visible inputs do not include ground-truth values; evaluation proceeds from each instance’s inputs and rules to `output/<id>/result.json`, followed by comparison against evaluator-only references. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone — no affirmative evidence supports a policy label; public identifiability was not externally verified.\n\n**Recommended Changes:**\nThe submission provides unusually thorough deterministic conventions, clean solver/reference separation, and strong numeric validation. No changes are required.","reviewed_at":"2026-08-20T09:24:36.543596Z"},"claim":null}