{"submission_id":"a214ca74-1cfc-49e3-ba75-05e45108e395","submission_revision_id":"16e6dde7-baeb-4d1b-b1e5-13174b5c502f","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-26T20:50:03.030283Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5 (3 sealed runs: 2 low reasoning, 1 medium reasoning)","harness":"Codex multi-agent","estimated_score":"<1%"}},"submitted_at":"2026-08-26T20:50:03.030283+00:00","industry_domain":"Education — Large-Scale Assessment, Writing Analytics, and AI Authorship Attribution","task_short_name":"pisa_fragmented_ai_passage_attribution_20_v1","operating_system":"Linux","task_description":"Objective: Identify the complete set of exact passages in 15-year-old students’ authentic PISA (Programme for International Student Assessment) writing responses that were generated by AI and transferred into the submitted text. Assessment context: This workflow uses original, unpublished argumentative writing and writing-process data from 15-year-old participants in the PISA 2025 study context. It represents an authentic large-scale educational assessment scenario in which evaluators see submitted texts but do not see the held-out writing histories and need to assess students’ performances. Starting state: The input package contains 20 held-out final texts. Every evaluation text contains 40–70% process-verified direct AI-transfer words and at least two separate AI-generated passages. It also contains a labeled, participant-disjoint training set from 40 additional students: 20 who wrote without an available chatbot and 20 with process-verified direct AI use. The training cases include final texts, complete decoded writing-process events, and known exact AI spans. Held-out process histories and gold labels are not included. Instructions and rules: 1. Analyze the labeled training texts and process histories. 2. For every held-out case, return all AI-generated passages as zero-based, half-open character spans. 3. Each submitted text field must exactly equal text[char_start:char_end]. 4. Spans may not overlap; include each of the 20 case IDs exactly once. 5. A held-out document is solved only when every hidden AI passage is recovered with both boundaries exactly correct and no passage is missing or added. Output: submission.jsonl following the supplied schema.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Validate the JSONL submission against the evaluator-only reference package.\n\nPrimary outcome: complete-document exact passage-set accuracy across 20 held-out documents. For a document to count as correct, the predicted set of (char_start, char_end) pairs must exactly equal the hidden gold set: every process-grounded AI passage must be present, both zero-based half-open boundaries must match exactly, and there must be no missing, extra, merged, split, overlapping, or off-by-one passage. The submitted text value for each span must equal the corresponding visible substring.\n\nFinal score = number of fully correct documents / 20.\n\nReport as diagnostics only:\n- micro strict exact-passage precision, recall, and F1 over 81 gold passages;\n- word-level AI and human F1;\n- tolerant passage F1; and\n- AI-share error.\n\nRequired controls:\n- perfect reference prediction = 1.0;\n- all-AI baseline = 0.0;\n- no-AI baseline = 0.0.\n\nProvenance verification: from the reference package root, run evaluator/reproduction/reproduce_process_gold.py. It must reproduce all 81 exact passages for all 20 cases. The package validator must confirm 20 evaluation participants, 40 disjoint training participants, balanced training classes, zero participant overlap, and no held-out gold/process leakage into the participant package.","software_and_version":"Python 3.9+; JSONL processing; no proprietary software required","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"934b890a-43cb-4e80-94ae-4e781f946a0f","filename":"pisa_writing_fragmented_authorship_ale_20_v1_participant.zip","category":"input_materials","content_type":"application/zip","size_bytes":4517675},{"file_id":"dc7a4062-8546-46d3-94c3-08e6075a0681","filename":"pisa_writing_fragmented_authorship_ale_20_v1_reference_and_provenance.zip","category":"reference_output","content_type":"application/zip","size_bytes":8500303},{"file_id":"a03efaf5-9b26-4fa1-b849-e594b8f6b126","filename":"pisa_writing_fragmented_authorship_ale_20_v1_gpt55_agent_solutions_only.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":32282},{"file_id":"29285a67-4c38-4ed4-9dd1-fb1fd484b716","filename":"selftest_primary_outcome_0_of_20.png","category":"difficulty_self_test_evidence","content_type":"image/png","size_bytes":389695}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, objectively evaluable, and supported by unusually thorough process-level provenance.\n\n**Completeness Check:**\n- Instruction: Complete — clearly defines the 20-case task, exact passage objective, zero-based half-open offsets, nonoverlap rules, and JSONL deliverable.\n- Inputs: Complete — includes 20 held-out texts and 40 balanced, participant-disjoint training cases with labels and complete decoded process histories.\n- Toolchain: Complete — Python 3.9+, standard-library JSON processing, and ordinary local archive utilities provide a feasible single-machine workflow.\n- Reference output: Complete — includes 81 exact gold passages, case summaries, token decisions, transfer audits, replay inputs, provenance ledgers, validation reports, scoring code, and reproduction code.\n- Rubric: Complete — the primary all-or-nothing document metric is precise, with strict passage, word-level, tolerant-passage, and AI-share diagnostics plus verified perfect, all-AI, and no-AI controls.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. All four manifest attachments were inspected: `pisa_writing_fragmented_authorship_ale_20_v1_participant.zip`, `pisa_writing_fragmented_authorship_ale_20_v1_reference_and_provenance.zip`, `pisa_writing_fragmented_authorship_ale_20_v1_gpt55_agent_solutions_only.zip`, and `selftest_primary_outcome_0_of_20.png`. Their formats and byte counts match the manifest; all archives are readable, unencrypted, bounded in expansion, and free of unsafe paths or links. Every structured file parsed successfully. Independent checks confirmed 20 unique evaluation cases, 40 balanced training cases, zero source-participant overlap, 81 valid nonoverlapping gold spans, exact substring consistency, the stated 40–70% AI-share range, and sealed participant/reference separation. All three self-test submissions are structurally valid and independently yield 0/20 exact documents, consistent with the screenshot.\n\nNo manifest attachment was left unreviewed. Large student-text and event-log contents were structurally validated and sampled rather than manually interpreting every event. Uploaded Python was inspected statically but not executed under safe-review rules; the packaged PASS reports and cross-file invariants independently support the claimed reproduction and controls. A solver can analyze the participant-visible training data, produce the required 20-row JSONL, and be evaluated deterministically against the sealed gold package on one ordinary machine.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone — the supplied evidence affirmatively describes authentic, unpublished assessment responses and process records rather than synthetic core data. Public identifiability was not independently searched outside the submitted evidence boundary.\n\n**Recommended Changes:**\nThe submission already provides a complete solver package, sealed evaluator package, process-level provenance, leakage controls, baseline controls, and consistent self-test evidence. No changes are required.","reviewed_at":"2026-08-26T21:00:22.622000Z"},"claim":null}