{"submission_id":"18b4e07d-9dd2-4f54-bbd1-6c757e23dd1a","submission_revision_id":"1c140662-18f9-46e7-a8a1-77c6a9264c69","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-27T06:22:46.143550Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5 (3 independent blind runs)","harness":"Codex independent agents; participant-only files; exact live description frozen by SHA-256","estimated_score":"<1%"}},"submitted_at":"2026-08-27T06:22:46.143550+00:00","industry_domain":"Education — Writing Assessment, Learning Analytics, and Feedback Uptake","task_short_name":"genius_ai_supported_writing_from_final_text_50_v2","operating_system":"Linux","task_description":"Objective: Inspect each submitted final student essay and identify, for every word, whether it belongs to writing performance produced independently by the student or the student supported by AI.\n\nProfessional Scenario: Schools increasingly use AI-generated formative feedback during writing instruction. Teachers usually receive a submitted final text, but they need to assess which parts reflect what a student could write independently and which parts were produced with AI support. A teacher-facing product analysis can make these two layers visible in the final response and support fair interpretation of independent writing competence, AI-supported performance, feedback uptake, and revision quality. \n\nStarting State: The agent receives 50 held-out authentic German argumentative essays submitted by students in grades 10 and 11 for a scientific study (https://doi.org/10.1016/j.lindif.2026.102979). Each student first completed an unsupported argumentative writing task and then completed the AI-feedback-supported task directly afterwards in the same classroom session. Because the two tasks were written immediately one after the other, substantial learning between them is not expected. (a) the immediately preceding independently written baseline essay on a structurally similar but different topic, (b) the submitted final AI-feedback-supported essay, and (c) a fixed word table with stable word IDs and exact half-open character offsets. The actual test feedback, local drafts, intermediate revisions, keystroke logs, and gold labels are hidden. Twenty student-disjoint worked examples disclose the complete writing and feedback evidence and correct word labels. \n\nInstructions and Rules: Create predictions.jsonl and classify every supplied final-text word exactly once as INDEPENDENT_WRITING or AI_SUPPORTED_WRITING. INDEPENDENT_WRITING means that the wording was already produced in the relevant local writing block without access to its AI feedback. AI_SUPPORTED_WRITING means that the wording first appeared while the student was revising with AI feedback available, including the final AI-feedback-supported holistic revision. The classification concerns the writing conditions under which the submitted product emerged. AI_SUPPORTED_WRITING does not mean that GPT-4o literally authored the word; the student entered the final wording. Use only the supplied word IDs. Do not retokenize the texts, change IDs, omit words, add words, or use any other labels.\n\nExpected Deliverable: predictions.jsonl containing exactly 50 case objects. Each object must contain the case_id and one prediction for every supplied word_id using only INDEPENDENT_WRITING or AI_SUPPORTED_WRITING.\n\nGround Truth: Hidden evaluator-side writing-process data establish exactly when each final-text word first appeared. Words preserved from the pre-feedback local drafts are labeled INDEPENDENT_WRITING. Words first introduced during AI-feedback-supported block revisions or the final holistic revision are labeled AI_SUPPORTED_WRITING. The evaluator package contains all five chronological draft-feedback-revision chains, final holistic revision logs, stable token offsets, deterministic alignment logic, gold labels, and source provenance. The test set contains 10,312 scored words. Every selected training and test case comes from the same 26–70% AI_SUPPORTED_WRITING eligibility range, contains at least 60 final words and all five feedback events, and has at least 0.80 alignment coverage. The absolute train-test mean label-share difference is 0.00003; the medians are 0.34279 and 0.34278.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Run evaluator/score_predictions.py predictions.jsonl. Primary success is exact-document accuracy. A student text counts as solved only if every supplied word_id occurs exactly once and every label exactly matches evaluator gold. Missing, extra, duplicated, malformed, unknown, or incorrectly labeled word IDs make that document incorrect. The official score is exact_documents / 50. Secondary diagnostics are overall word accuracy, per-label F1, macro-F1, and exact transition-boundary F1; they do not replace the primary metric. The perfect reference scores 50/50. Empty, all-INDEPENDENT_WRITING, and all-AI_SUPPORTED_WRITING submissions each score 0/50 exact documents. The evaluator package contains gold.jsonl, process_evidence.jsonl, source_provenance.jsonl, scoring code, frozen controls, hashes, validation reports, the deterministic distribution-matched split, and the reproducible corpus builder.","software_and_version":"Python 3.11+; UTF-8 JSONL; standard-library evaluator","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"db6a195e-9fcd-4c49-92f8-caa41fae0511","filename":"genius_word_provenance_ale_50_v1_participant.zip","category":"input_materials","content_type":"application/zip","size_bytes":258358},{"file_id":"96054b67-51ab-4e34-9320-f8d7a2a13a7d","filename":"genius_ai_supported_writing_50_v2_gpt55_selftest.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":60144},{"file_id":"655235ea-e3e6-4121-9a53-3378226869eb","filename":"selftest_primary_outcome_0_of_50.png","category":"difficulty_self_test_evidence","content_type":"image/png","size_bytes":172139},{"file_id":"c71e4160-03d3-4c8a-bcee-77f553e1fac3","filename":"genius_word_provenance_ale_50_v1_reference_and_provenance.zip","category":"reference_output","content_type":"application/zip","size_bytes":905099}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The revision is complete, safely reachable, professionally grounded, and objectively scorable, with only negligible packaging and validation-hardening issues.\n\n**Completeness Check:**\n\n- Instruction: Complete — The classification construct, two permitted labels, word-ID constraints, and exact JSONL deliverable are explicit.\n- Inputs: Complete — Fifty test cases contain 10,312 sequential words with valid offsets; 20 disjoint worked examples contain five complete feedback events and gold labels.\n- Toolchain: Complete — Python 3.11+, UTF-8 JSONL, Linux, and a standard-library evaluator provide a practical single-machine workflow.\n- Reference output: Complete — Sealed gold labels, chronological process evidence, provenance records, deterministic derivation logic, controls, and audit documentation cover all test cases.\n- Rubric: Complete — Exact-document accuracy is unambiguous, with word accuracy, per-label F1, macro-F1, and boundary F1 as secondary diagnostics.\n\n**File and Task Reachability:**\n\nAttachment classification: reachable. Task classification: reachable.\n\nAll four attachments were inspected: `genius_word_provenance_ale_50_v1_participant.zip`, `genius_ai_supported_writing_50_v2_gpt55_selftest.zip`, `selftest_primary_outcome_0_of_50.png`, and `genius_word_provenance_ale_50_v1_reference_and_provenance.zip`. Their byte counts and actual formats match the manifest. The ZIPs are readable, unencrypted, CRC-valid, bounded in expansion, and contain only safe relative regular-file and directory entries. UTF-8 JSON/JSONL content parses successfully; schemas, counts, offsets, labels, controls, scorer logic, provenance, and representative records were inspected. The 1600×1600 PNG is readable and visually agrees with the self-test results.\n\nThe three self-test outputs each contain all 50 cases and 10,312 labels. Their independently recomputed exact-document and word-accuracy results match the submitted reports. The claimed blindness is procedural rather than filesystem-enforced, but this is adequately disclosed.\n\nOne internal manifest digest differs for the empty-control score because two zero-valued label fields were serialized in the opposite order; the content is semantically identical and all metrics agree. The scorer also does not reject additional unknown case objects once all required cases are valid, although it correctly rejects missing, duplicate, extra, or invalid word IDs within scored cases. Neither issue obstructs completion or objective scoring.\n\nThe end-to-end path is complete: inspect the participant cases and worked examples, fill the supplied prediction template, write `predictions.jsonl`, and evaluate it against the sealed gold with the supplied scorer. No attachment content remained materially unreviewed, and there were no material tooling limitations.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\n- `public_benchmark` — The submission affirmatively states that the student texts were previously published under CC BY 4.0. Risk is substantially mitigated because the process traces, selected-case provenance, and exact word labels are non-public and evaluator-sealed; public essay retrieval would not reveal the protected answers.\n\n**Recommended Changes:**\n\nThe submission already provides unusually strong participant/evaluator separation, process-backed labels, controls, provenance documentation, and reproducible self-test evidence. No material changes are required. Optional hardening would be to regenerate the internal manifest only after final packaging, make label-field serialization deterministic, and have the scorer reject unknown or additional case objects so it enforces the stated exactly-50-object contract literally.","reviewed_at":"2026-08-27T06:33:32.184755Z"},"claim":null}