{"submission_id":"c035cfdc-f6bf-4ba0-a1e6-1123ed7a95e9","submission_revision_id":"59182c13-7c6a-4267-b71e-f93942a0ba2f","revision_version":1,"submission_status":"under_review","submitted_at":"2026-09-27T22:53:29.048600Z","submitter_name":"Felix Juefei-Xu","submitter_email":"juefei.xu@gmail.com","task":{"status":"submitted","version":1,"extra_fields":{},"submitted_at":"2026-09-27T22:53:29.048600+00:00","industry_domain":"Scientific Research Integrity / Biomedical Image Forensics","task_short_name":"sci_integrity_image_forensics_v1","operating_system":"Linux","task_description":"Objective: Conduct an end-to-end forensic integrity audit of a blinded corpus of biomedical scientific figures. Identify image regions showing suspicious duplication or reuse, rank studies by review priority, localize the supporting evidence, classify each finding, and produce a structured audit report.\n\nStarting State: The input archive contains 10 anonymized studies with 60 scientific figure sheets in total, along with blinded study context, generic figure descriptions, an anomaly ontology, task instructions, and an exact output schema. Study identities, authors, titles, DOIs, journal outcomes, retractions/corrections, and post-publication discussions are not provided.\n\nInstructions & Rules:\n\nReview all 60 figure sheets across the 10 studies.\n\nIdentify image duplication or reuse that warrants expert integrity review. Compare figures both within and across a study when appropriate.\n\nFor every reported finding, identify the two implicated figure images and provide normalized bounding boxes [x1, y1, x2, y2] for the corresponding source regions.\n\nClassify each finding using the supplied anomaly ontology and assign an evidence-calibrated severity and confidence.\n\nReport forensic similarity evidence, including the best geometric transformation when applicable and similarity measurements such as SSIM, normalized cross-correlation, or feature matching.\n\nRank all 10 studies by review priority.\n\nProduce all outputs required by output_schema.json, including:\n\npaper_ranking.csv\n\nfindings.jsonl\n\nevidence_regions.jsonl\n\nevidence_metrics.jsonl\n\naudit_report.md\n\nControl false positives. Do not report visual similarity unless the localized regions provide meaningful forensic evidence.\n\nDo not infer author intent or accuse any individual of fraud. Report observable image-integrity evidence only.\n\nDo not use external web search, reverse-image search, publication databases, or other internet sources. The task must be solved solely from the supplied frozen input materials.\n\nThe task is designed to test scientific image-forensics reasoning rather than recognition of historically known papers. Study and figure identifiers are randomized and figures have been freshly transformed while preserving the forensic relationships required to solve the task.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Use the deterministic evaluator included in the reference package.\n\nThe agent output is evaluated against hidden finding-level ground truth and, critically, against the pixels in the staged input images rather than trusting similarity values claimed by the agent.\n\n1. Output validation\n\nVerify that all required files exist and conform to the supplied schemas:\n\npaper_ranking.csv\n\nfindings.jsonl\n\nevidence_regions.jsonl\n\nevidence_metrics.jsonl\n\naudit_report.md\n\nMalformed records, invalid figure IDs, invalid bounding boxes, or missing required fields receive no credit for the affected component.\n\n2. Study ranking — 1 point\n\nCompare the submitted ranking of the 10 anonymized studies with the hidden reference ranking.\n\nMaximum: 1/100\n\n3. Finding identification — 2 points\n\nMatch submitted findings to the hidden verified findings using anonymized study IDs and figure-pair identities.\n\nThe frozen reference contains 6 verified positive findings.\n\nMaximum: 2/100\n\n4. Classification — 1 point\n\nEvaluate anomaly type and severity against the hidden reference ontology.\n\nMaximum: 1/100\n\n5. Forensic evidence validity — 92 points\n\nThis is the primary evaluation component.\n\nFor each claimed finding:\n\nRead the submitted normalized bounding boxes.\n\nCrop those regions directly from the frozen staged input figures.\n\nReject invalid evidence including:\n\nout-of-range coordinates;\n\nextremely small crops;\n\nnearly uniform crops;\n\ntrivial label/background matches;\n\ninvalid self-overlap.\n\nRecompute similarity independently rather than trusting the agent's reported metrics.\n\nTest common evidence-preserving transformations including identity, horizontal/vertical flips, and 90-degree rotations.\n\nRecompute structural and pixel/feature similarity using SSIM, normalized cross-correlation, and ORB-based feature matching.\n\nApply the finding-specific thresholds contained in the hidden evaluation package.\n\nMaximum: 92/100\n\n6. False-positive and severity calibration — 4 points\n\nPenalize unsupported findings and inappropriate severity escalation, including overcalling the lower-severity calibration case.\n\nMaximum: 4/100\n\nFinal score\n\nTotal = ranking + finding identification + classification + forensic evidence + false-positive calibration\n\nMaximum: 100\n\nThe frozen expert reference solution scores approximately 99.798/100, with:\n\n6 true positives\n\n0 false positives\n\n0 false negatives\n\nprecision = 1.0\n\nrecall = 1.0\n\nF1 = 1.0\n\nAs an anti-shortcut check, removing valid localized forensic evidence reduces the attainable reference score to approximately 8/100. Therefore, recognition or memorization of the source studies is insufficient to obtain a high score.\n\nThe evaluator should run with general outbound internet access disabled during the solve phase.","software_and_version":"Python 3.11; OpenCV 4.x; scikit-image 0.24+; NumPy 2.x; SciPy 1.x; Pillow 10+","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"867228ca-d6b2-459b-ab6a-1ac57c0d036d","filename":"SciIntegrityBench_ALE_OUTPUT_FINAL_v2.zip","category":"reference_output","content_type":"application/zip","size_bytes":27078},{"file_id":"7b182314-65dd-4a88-8bd2-d86c67df4203","filename":"SciIntegrityBench_ALE_INPUT_FINAL.zip","category":"input_materials","content_type":"application/zip","size_bytes":163097874}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally constructed, objectively evaluable, and reachable end to end with strong provenance and anti-shortcut controls.\n\n**Completeness Check:**\n- Instruction: Complete — Defines the corpus, forensic objective, evidence standard, prohibited conduct, required outputs, and offline constraint.\n- Inputs: Complete — Includes 60 valid RGB PNG figure sheets across 10 studies, consistent manifests and captions, study context, ontology, and output schema.\n- Toolchain: Complete — Uses legally obtainable open-source Python image-analysis libraries on Linux with compatible version requirements.\n- Reference output: Complete — Supplies six consistent findings, normalized evidence regions, metrics, ranking, report, provenance, hashes, and evaluator source.\n- Rubric: Complete — Allocates all 100 points, emphasizes independently recomputed pixel evidence, and explicitly handles false positives and severity calibration.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`. Both `SciIntegrityBench_ALE_INPUT_FINAL.zip` and `SciIntegrityBench_ALE_OUTPUT_FINAL_v2.zip` were inspected. They are valid, unencrypted ZIP archives of the declared sizes, contain 75 and 21 regular-file members respectively, pass archive integrity checks, have no unsafe paths, and show no excessive expansion.\n\nThe input task, ontology, corpus manifest, captions, output schema, all study-context files, and all 60 PNGs were inspected structurally; representative positive and control figure sheets were also inspected visually. All image identifiers align across the manifest, captions, and reference hashes, and all 60 staged-image hashes match. The reference JSON, JSONL, CSV, Markdown, and Python source files were inspected, including the validation, scoring, forensic-comparison, provenance, attribution, transform, ground-truth, and anti-shortcut materials. The redundant compiled Python cache was inventoried but not executed or semantically relied upon. The reviewer environment lacked Python, so the uploaded evaluator was source-reviewed rather than rerun; this is a reviewer-tooling limitation, not a submission defect.\n\nThe completion path is clear: inspect the staged images offline, emit the five schema-defined artifacts, validate their structure, and score claimed regions by recropping the frozen images and recomputing transform-aware similarity. Reference records contain six unique findings with valid boxes and numeric metrics, and the supplied reference score is consistent with the stated rubric. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The underlying published figures and later verification notices are publicly sourced according to the submitted provenance. Risk is materially mitigated by private randomized aliases, fresh pixel transformations, an offline solver, matched controls, and a rubric assigning 92% of credit to independently verified localization on the transformed pixels. No affirmative evidence supports `ai_synthetic` or `synthetic_defect_fixture`.\n\n**Recommended Changes:**\nThe submission provides strong provenance, complete solver inputs, calibrated ground truth, and a defensible behavioral evaluator. No changes are required.","reviewed_at":"2026-09-28T01:00:44.770520Z"},"claim":null}