{"submission_id":"e0b94d21-1987-4d02-92ac-31ff9e64ad47","submission_revision_id":"f8e75373-4c9d-49e0-af62-e4ce9e8cf4b7","revision_version":2,"submission_status":"submitted","submitted_at":"2026-07-30T19:02:03.376486Z","submitter_name":"Shengrui Lyu","submitter_email":"lsr@anthropic.com","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"40–60%"}},"submitted_at":"2026-07-30T19:02:03.376486+00:00","industry_domain":"Geospatial / GIS (Cartography and Historical Geography)","task_short_name":"gis_historical_topo_terrain_1","operating_system":"Linux","task_description":"Objective: Reconstruct the 1957 land surface of a Kansas river valley from four scanned historical topographic maps: georeference the sheets from their printed collars, digitize the printed contour lines over a defined study area, and interpolate a historical elevation model (DEM) from those contours.\n\nStarting State: The agent (a GIS specialist working in QGIS 3 on Ubuntu, with GDAL available) receives four adjacent 1957 USGS 1:24,000 quadrangle scans (`input/sheets/sheet_01.tif` .. `sheet_04.tif` — Westfall SW, Westfall SE, Ellsworth, Carneiro; Ellsworth County, Kansas) delivered as plain images with ALL georeferencing removed; the printed map collar (corner latitude/longitude labels, 2.5-minute graticule ticks, UTM ticks, projection/datum notes, contour interval) is intact. Also provided: `input/task_brief.md` (which defines the Area of Interest as a pixel-space polygon on sheets 03 and 04, so no coordinates are leaked) and `input/schema.md` (exact output specification). No internet access is permitted (the sheets are public archives, so the sandbox must be offline).\n\nInstructions & Rules:\n1. Georeference each of the four sheets by reading the control coordinates off the printed collar; deliver each as a GeoTIFF in EPSG:26714 (NAD27 / UTM zone 14N) plus a CSV of the ground control points used (`output/georef/sheet_NN.tif`, `output/georef/sheet_NN_gcps.csv`).\n2. Digitize every contour line printed inside the Area of Interest into `output/contours.gpkg` (layer `contours`, LineString, integer elevation-in-feet attribute `elev_ft`), reading elevations from the printed contour labels, index contours and spot heights; join lines across label gaps; do not digitize contours across open water.\n3. Interpolate a historical DEM from the contours (spot heights optional): `output/historical_dem.tif`, a 10 m elevation raster in feet, EPSG:26714, covering the whole Area of Interest.\n4. Save a QGIS project loading the georeferenced sheets, contours and DEM (`output/topo_project.qgz`) and a short `output/method_note.md`.\nScoring rewards positional accuracy of the georeferencing (metres), contour completeness and elevation correctness, and DEM elevation error against independent modern lidar terrain (open water excluded). Terrain a full contour interval off, or a DEM built from spot heights instead of the contour lines, does not score.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Deterministic, offline grader (Python; grade_georef.py supplied in the reference bundle). The hidden reference is authoritative public-domain data, not a hand-labelled key: (a) the original geotransforms of the four USGS sheets (removed from the agent copies), (b) USGS 3DEP 1/3-arc-second lidar terrain reprojected to EPSG:26714 in NGVD29 feet, (c) an Area-of-Interest polygon and a water/reservoir mask.\n\nHard gates (score 0.0): missing/unreadable georeferenced rasters or GCP files; contours GeoPackage missing, no `contours` layer or no `elev_ft` field; DEM missing, unreadable, without CRS/geotransform, or not covering the AOI.\n\nScore = 0.20 * georeferencing + 0.50 * terrain + 0.25 * contours + 0.05 * project, each in [0,1]:\n- Georeferencing: per sheet, RMSE in ground metres of the delivered GCPs against the reference transform plus an FFT phase-correlation registration of the delivered raster against the original; curve 10 m -> 1.0, 25 m -> 0.7, 50 m -> 0.4, else 0.\n- Terrain (dominant): resample the delivered DEM to the reference grid over AOI minus water; remove a global vertical bias capped at 2.5 ft (0.25 contour interval, so a systematic one-interval mis-attribution is NOT forgiven); de-biased RMSE curve <=5.5 ft -> 1.0, <=8 -> 0.7, <=12 -> 0.4, <=20 -> 0.15, else 0; multiplied by a DEM-coverage factor and a contour-support factor (a DEM must be supported by digitized contour lines nearby, so a smuggled or spot-height-only DEM caps around 0.25).\n- Contours: 0.5 * credible-line coverage (fraction of terrain cells within 150 m of a line whose vertices agree within one contour interval with 3DEP, so fabricated dense line webs earn no coverage) + 0.3 * along-line elevation agreement vs 3DEP + 0.2 * attribute sanity (values are multiples of the interval within a plausible range).\n- Project: valid .qgz referencing the expected layers.\n\nCalibration/fixtures (all reproducible): the expert reference solution scores 1.000 (measured de-biased RMSE 4.21 ft over the full AOI); empty output 0.0; a perfect 3DEP DEM submitted with no contours 0.25; a DEM interpolated from ~40 spot heights 0.25; everything offset by one contour interval 0.66; DEM over half the AOI 0.50; index contours only 0.50. A regression suite (103 tests) pins these values, hard gates, determinism and anti-fabrication cases.\n\n---------------------------------------------------------------------------------------\nEdit for addressing reviewer feedback (also re-uploaded reference) bundle): review found that a digitized line with fewer than 3 in-region samples was classed 'unverifiable' yet still counted toward credible coverage, so a dense grid of short two-vertex fragments could unlock full contour support - and hence terrain credit - for an otherwise unsupported DEM (the unfixed grader scored that construction 1.00). Fixed as follows. (1) Coverage and elevation-agreement now credit ONLY verifiably good lines: a line is classifiable only with >= 3 in-region verification samples AND >= 40 m of in-region length; unverifiable and bad lines earn nothing. (2) New regression fixture neg_fragment_grid = the exact attack (perfect DEM + ~20 000 short, individually correctly-labelled two-vertex fragments) now scores 0.30 - georeferencing + project + attribute sanity only (coverage 0, agreement 0, terrain 0), pinned by test at <= 0.30. (3) pytest added to environment.yml so 'make test' runs from a clean environment; the suite is now 103 tests, all passing. On the length threshold: ~100 m was evaluated against the expert reference (median honest vectorizer piece = 67 m, broken at labels and roads) and would have penalized correct solutions; 40 m is the length carrying three verification samples and defeats the fragment grid identically. All other fixture scores unchanged except neg_index_only (0.503 -> 0.500). Details: EVALUATION.md and README.md 'Revision note (reviewer feedback)'.","software_and_version":"QGIS 3.34 LTR; GDAL >= 3.4; Python 3 with numpy + scipy (grading only)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"1cbeb31b-fd21-4413-a480-0c6f14983be9","filename":"historical_topo_terrain_input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":35136373},{"file_id":"7918fe0d-aab1-4364-91d8-c7834ed80d05","filename":"historical_topo_terrain_difficulty_evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":4873},{"file_id":"0bcb4dcc-46ca-45ab-bc8a-ddedfc1fd327","filename":"historical_topo_terrain_reference_and_evaluation.zip","category":"reference_output","content_type":"application/zip","size_bytes":361142838}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The revised submission is complete, professionally scoped, end-to-end reachable, and supported by a well-calibrated evaluator that closes the reported contour-fragment loophole.\n\n**Completeness Check:**\n- Instruction: Complete — The AOI, workflow, CRS, units, output paths, schemas, and quality expectations are explicit and internally consistent.\n- Inputs: Complete — Four valid unreferenced TIFF scans and the solver-facing brief and schema are supplied; low-level TIFF inspection confirms dimensions and removal of GeoTIFF positioning tags.\n- Toolchain: Complete — QGIS, GDAL, and the open-source Python grading stack provide a viable offline workflow on the specified single-machine environment.\n- Reference output: Complete — The bundle contains original georeferenced sheets, reference transforms, a 10 m terrain raster, AOI and water-mask layers, and a complete expert output scoring 1.000.\n- Rubric: Complete — Hard gates, component weights, scoring curves, coverage factors, and partial-credit behavior are objective and documented. Positive and negative fixtures cover empty, shifted, offset, partial, index-only, spot-height, unsupported-DEM, and fragment-grid cases.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. The inspected files were `historical_topo_terrain_input_materials.zip`, `historical_topo_terrain_difficulty_evidence.zip`, and `historical_topo_terrain_reference_and_evaluation.zip`. All matched their manifest byte counts, passed full container integrity checks, were unencrypted, and contained no unsafe paths or special filesystem entries. No manifest attachment was left unreviewed. Representative TIFF tags, Markdown specifications, JSON reports, GeoPackage schemas, QGIS project XML, provenance, evaluator logic, and regression definitions were inspected. Uploaded programs were reviewed statically and not executed. The review environment lacked GDAL/OGR and Python, so the submitted 103-test suite could not be rerun and full TIFF pixels could not be rendered; this is a reviewer-tooling limitation, not a submission defect. Static test enumeration supports the stated count, and recorded reports match the documented fixture outcomes. The solver can georeference the supplied scans, digitize contours, interpolate the required DEM, create the project, and be evaluated entirely offline. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The source map sheets and modern terrain reference are explicitly identifiable public USGS products. Direct retrieval could expose georeferencing and terrain answers, but the mandatory offline solver environment materially mitigates that route; the bespoke contour digitization is not shown to be a canonical memorized benchmark.\n\n**Recommended Changes:**\nThe revision validly fixes the fragment-grid vulnerability by requiring at least three in-region samples and 40 m of in-region length, crediting verified-good lines only, and pinning the attack at 0.30. No changes are required. Optional documentation cleanup could reconcile the positive fixture method note’s claim of 16 GCPs per sheet with the supplied nine-row CSVs and remove a few stale or duplicated bundle-path references.","reviewed_at":"2026-08-14T01:32:55.226373Z"},"claim":null}