{"submission_id":"bc2730d1-376e-4b55-ba4e-96d57f02aece","submission_revision_id":"b10f154d-8904-4cfb-bf2f-d78956b5ae53","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-30T07:58:08.637179Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5 (medium and low reasoning)","harness":"Codex blind full-benchmark runs; official offline OCI scorer","estimated_score":"<1%"}},"submitted_at":"2026-08-30T07:58:08.637179+00:00","industry_domain":"Scientific Research, Quantitative Data Analysis, and Data Visualization","task_short_name":"scientific_figure_transfer_procedural_twin_50_v1","operating_system":"Linux","task_description":"Researchers frequently encounter a useful scientific figure and want to determine what the same analysis and visualization would look like with their own data. To make resulting figures directly comparable, differences should reflect the data rather than unintended changes in analysis, scales, layout, styling, or rendering.\n\nFor each of 50 cases, the agent receives a visible synthetic reference figure, independently generated synthetic target data, and a de-identified methods card. It must infer and implement the complete case-specific analytical and graphical procedure in R, recompute every value from the supplied target data, and reproduce the hidden target-data gold exactly. It may not copy, embed, trace, rename, edit, or return the reference image.\n\nEvery case requires solution.R, figure.png, figure_data.csv, and run_manifest.json. The R program must implement Rscript --vanilla solution.R INPUT_DIR OUTPUT_DIR. Evaluation reruns it in clean offline staging in the locked Linux AMD64 runtime without access to reference data, canonical code, or gold.\n\nAll participant data are newly generated synthetic records released under CC0-1.0. Participant materials contain no article title, DOI, repository URL, public-study data, original filename, historical script, or publicly searchable target. No external materialization is required.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The benchmark has 50 cases and 100 binary points. Each case contributes one code/data point when the submitted R program executes in clean offline staging, exactly recreates its submitted outputs, and produces canonical plotted data equal to evaluator gold. It contributes one image point only when the submitted PNG has the correct dimensions and every decoded RGBA pixel equals gold. Pixel similarity below exact equality receives no partial credit. Divide the 0–100 total by 100 if ALE requires a score in the interval from zero to one.\n\nThe complete positive control must score 100/100 and the empty negative control 0/100. Official gold and all submission replays use the same locked Linux AMD64 image with networking disabled.","software_and_version":"R 4.5.1; ggplot2 4.0.3; ragg 1.5.2; pinned Linux AMD64 OCI runtime","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"78b2a507-3854-4804-8b4e-d1314f58ce99","filename":"01_ScientificFigureTransfer_ProceduralTwin50_Participant_Input.zip","category":"input_materials","content_type":"application/zip","size_bytes":2035610},{"file_id":"f0d561f8-902b-440a-8b13-88392cd820c4","filename":"03_ScientificFigureTransfer_ProceduralTwin50_Validation_Evidence.zip","category":"reference_output","content_type":"application/zip","size_bytes":36917},{"file_id":"1d20ec23-9a11-4366-9c07-3d68ea874051","filename":"02_ScientificFigureTransfer_ProceduralTwin50_Hidden_Gold_and_Evaluator.zip","category":"reference_output","content_type":"application/zip","size_bytes":462942869},{"file_id":"a7510226-51b2-49e5-b3e5-31c00d864526","filename":"04_ScientificFigureTransfer_ProceduralTwin50_GPT55_Blind_Test_and_Image_Comparisons.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":14891512}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally scoped, reproducible in a supplied offline runtime, and supported by coherent inputs, hidden gold, deterministic controls, and an objective case-level rubric.\n\n**Completeness Check:**\n- Instruction: Complete — Defines the 50-case analytical transfer task, prohibited reference-image reuse, four required artifacts per case, and the exact R command interface.\n- Inputs: Complete — All 50 cases contain readable target CSV data, methods context, licensing, and a synthetic reference PNG; ten procedure families and claimed row counts were verified.\n- Toolchain: Complete — Supplies a locked Linux AMD64 OCI image with R 4.5.1, pinned package versions, fonts, locale, timezone, and offline execution requirements.\n- Reference output: Complete — Contains 50 hidden gold PNGs, 50 canonical plotted-data CSVs, canonical R programs, reference and target inputs, deterministic replays, and positive and negative controls.\n- Rubric: Complete — Awards 100 binary points across exact canonical data and decoded-RGBA image equality, with executable scorer logic and verified 100/100 and 0/100 controls.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. The inspected files were `01_ScientificFigureTransfer_ProceduralTwin50_Participant_Input.zip`, `03_ScientificFigureTransfer_ProceduralTwin50_Validation_Evidence.zip`, `02_ScientificFigureTransfer_ProceduralTwin50_Hidden_Gold_and_Evaluator.zip`, and `04_ScientificFigureTransfer_ProceduralTwin50_GPT55_Blind_Test_and_Image_Comparisons.zip`. All match their declared ZIP format and manifest size, pass archive CRC validation, and contain only regular, unencrypted members without unsafe paths. The supplied OCI tar has a standard image layout and matches its declared SHA-256. Representative methods, data schemas, figures, gold outputs, scorer logic, validation reports, controls, blind-run results, licensing, and runtime manifests were inspected. Repeated generated images and all 100 blind-run R programs were not individually reviewed line by line; no material content was inaccessible and no tooling limitation affected the assessment. A solver receives only the participant materials, produces four artifacts for each case, and is rerun against sealed target inputs in the offline OCI runtime before canonical CSV and exact pixel comparison. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission already provides a coherent professional workflow, strong solver/evaluator separation, complete runtime materialization, and unusually thorough validation evidence. No changes are required.","reviewed_at":"2026-08-30T08:07:54.178816Z"},"claim":null}