{"claim":{"id":"473da79c-2b5c-44d4-9e2b-ce22e1d978fe","submission_id":"c772ff4d-8754-4b5f-87ad-5f2454fe09f4","submission_revision_id":"16c233d4-6113-47fd-9353-ac31d4bf6232","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:14:12.390960Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"c772ff4d-8754-4b5f-87ad-5f2454fe09f4","submission_revision_id":"16c233d4-6113-47fd-9353-ac31d4bf6232","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-29T21:33:39.639178Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5 (two independent runs, medium reasoning)","harness":"Codex agent harness","estimated_score":"<1%"}},"submitted_at":"2026-08-29T21:33:39.639178+00:00","industry_domain":"Scientific Research, Quantitative Data Analysis, and Data Visualization","task_short_name":"scientific_figure_transfer_open50_v1","operating_system":"MacOS","task_description":"# Scientific Figure Transfer: Transferring a figure from an article to to new data\n\n## Professional context\n\nResearchers frequently encounter a useful figure in another scientific article and want to determine what the same analysis and visualization would look like with their own data. To make the resulting figures directly comparable, differences should reflect the data rather than unintended changes in analysis, scales, layout, styling, or rendering.\n\n## Task goal\n\nThe task is to reconstruct 50 scientific figures from 29 open-access articles. For each case, the visible synthetic figure represents the figure encountered in an article, while the supplied public original study data represent the new data to which the procedure must be transferred. The agent also receives a methods excerpt or methods context.\n\nFor every case in input/, write and execute case-specific R code that reproduces the scientific figure shown in synthetic_reference.png, but uses the supplied data files as the new analysis input. Infer and implement the complete analytical and graphical procedure: analytical transformations, statistical summaries, plotted values, geometry, panel structure, scales, axes, labels, colors, typography, dimensions, and rendering choices.\n\nThe synthetic reference was generated with the same visualization procedure but different, signal-rich synthetic data. Recompute the figure from the supplied data. Do not copy, embed, trace, rename, edit, or return the reference image. Use the methods excerpt to recover the article's analytical decisions.\n\nFor every case, create exactly:\n\n- solution.R\n- figure.png\n- figure_data.csv\n- run_manifest.json\n\nsolution.R must implement:\n\nRscript --vanilla solution.R INPUT_DIR OUTPUT_DIR\n\nIt must read the analysis inputs from INPUT_DIR and deterministically recreate figure.png and figure_data.csv in OUTPUT_DIR. The evaluator reruns the program in a clean workspace without access to synthetic_reference.png.\n\nThe participant inputs are fully provenance-controlled. All 29 articles are open access. Every analysis input is either retrieved from an openly accessible original study repository or is one of two author-owned files released under CC BY 4.0. PUBLIC_SOURCE_MANIFEST.csv records filenames, source and repository URLs, redistribution status, byte sizes, and SHA-256 hashes. Where a public repository provides no explicit redistribution license, materialize_public_inputs.py retrieves the file from its original public source and verifies its pinned SHA-256; it is not republished in the participant archive.\n\nBecause decoded-pixel identity is scored, official gold generation and evaluation must use the same pinned R, graphics device, packages, fonts, and operating-system environment recorded in evaluator/runtime/. The task succeeds only when the R program recovers the exact canonical data represented in the original-data figure and the generated PNG exactly matches the freshly rendered evaluator gold image.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Run the sealed evaluator over all 50 case directories. The benchmark produces 100 binary points, exactly two points per case:\n\n1. One R-code point if solution.R parses and executes successfully in an independently staged clean workspace, recreates the submitted figure and canonical plotted data, and produces a figure_data.csv that exactly equals the evaluator gold.\n2. One image point if the submitted PNG has the correct dimensions and every decoded RGBA pixel exactly equals the evaluator gold image.\n\nThus exact reproducible R code contributes 50 points and exact images contribute 50 points. Pixel similarity below exact identity earns no partial credit. Input-file provenance and SHA-256 agreement remain diagnostic and do not directly contribute points. If ALE requires a score in [0,1], divide the 0–100 total by 100.\n\nA perfect submission contains all four required files for all 50 cases. Every solution.R runs in clean staging, depends only on the supplied analysis inputs, reproduces the exact evaluator canonical figure data, and generates a PNG identical to gold in dimensions and every decoded RGBA pixel. Missing files, non-executable code, hard-coded reference values, copying synthetic_reference.png, reliance on the input directory's basename, or failure in clean staging loses the corresponding binary point. An empty submission scores 0/100.\n\nValidation evidence: participant, evaluator, and gold case sets agree for all 50 cases; 206/206 structural and leakage checks pass; all 25 newly added Atlas cases pass clean exact replay; the complete positive control scores 100/100; and the empty negative control scores 0/100. Two independent blind GPT-5.5 medium-reasoning runs scored 0/100 and 0/100, while the positive control demonstrates that a perfect score is technically reachable.","software_and_version":"R 4.5.1; Python 3.11+ for deterministic public-input preparation; system terminal or text editor","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"f4727065-c1d4-4605-8bff-f79b2450e4f2","filename":"01_ScientificFigureTransfer_Open50_Participant_Input.zip","category":"input_materials","content_type":"application/zip","size_bytes":44355231},{"file_id":"2cb97f74-5272-4c3a-869c-be827892218f","filename":"02_ScientificFigureTransfer_Open50_Hidden_Gold_and_Evaluator.zip","category":"reference_output","content_type":"application/zip","size_bytes":57399062},{"file_id":"72e4612d-25c2-40b2-a505-58dd3da52fea","filename":"03_ScientificFigureTransfer_Open50_GPT55_Blind_SelfTest_Evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":17646023}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, professionally scoped, objectively scored, and operationally reachable, with public-answer lookup risk that can be mitigated through offline solver execution.\n\n**Completeness Check:**\n- Instruction: Complete — The 50-case workflow, prohibited shortcuts, required R interface, outputs, and deterministic reconstruction goal are explicit.\n- Inputs: Complete — All cases include synthetic references and methods context; 68 input instances are checksum-pinned, with 15 bundled and 53 retrieved through a verified materialization process.\n- Toolchain: Complete — R, Python, macOS architecture/version, rendering devices, and principal package versions are documented; exact scoring requires the stated pinned environment.\n- Reference output: Complete — All 50 cases have gold PNGs, canonical CSV data, provenance manifests, and evaluator-only reference code.\n- Rubric: Complete — The scorer implements 50 binary code points and 50 decoded-RGBA image points, with positive and empty controls reporting 100/100 and 0/100.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. I inspected `01_ScientificFigureTransfer_Open50_Participant_Input.zip`, `02_ScientificFigureTransfer_Open50_Hidden_Gold_and_Evaluator.zip`, and `03_ScientificFigureTransfer_Open50_GPT55_Blind_SelfTest_Evidence.zip`. All three are valid, unencrypted ZIP archives without traversal paths or links; their internal file counts, sizes, and SHA-256 manifests are consistent. Inspection covered the task and output contracts, source and article manifests, materialization and evaluator-preparation code as inert text, case inventories, methods PDFs, representative CSV/XLSX schemas, PNG dimensions and visual samples, gold manifests, scorer logic, runtime records, validation reports, and blind-run evidence. Uploaded programs were not executed, serialized R data were signature-checked rather than deserialized, and the 53 source-fetched instances were not independently retrieved under the evidence boundary. End to end, setup materializes and verifies public inputs, the solver produces four artifacts for each case, evaluator setup stages identical inputs, and the sealed scorer reruns each R program and compares canonical data and decoded pixels. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The source articles, study data, repository provenance, and some historical analysis filenames are publicly identifiable, so external lookup could expose substantial parts of the target procedure. Materialize inputs before the solver starts, then disable external network access and keep evaluator assets sealed.\n\n**Recommended Changes:**\nThe submission provides coherent contracts, strong provenance records, complete gold coverage, and meaningful controls. No contributor changes are required. Administrators should enforce offline solver execution after input materialization and use the pinned macOS rendering environment for official scoring.","reviewed_at":"2026-08-29T21:49:28.542325Z"},"claim":{"id":"473da79c-2b5c-44d4-9e2b-ce22e1d978fe","submission_id":"c772ff4d-8754-4b5f-87ad-5f2454fe09f4","submission_revision_id":"16c233d4-6113-47fd-9353-ac31d4bf6232","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:14:12.390960Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}