{"claim":{"id":"5f297a6a-f5c1-4108-a752-85c6e2eae150","submission_id":"4fce97e8-3811-49b9-b859-a5d6d8c2a595","submission_revision_id":"5f6281cc-d5b5-4df5-9c15-b9d3b4d8a757","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:12:57.489373Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"4fce97e8-3811-49b9-b859-a5d6d8c2a595","submission_revision_id":"5f6281cc-d5b5-4df5-9c15-b9d3b4d8a757","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-27T20:01:48.188730Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5","harness":"Codex agent","estimated_score":"<1%"}},"submitted_at":"2026-08-27T20:01:48.188730+00:00","industry_domain":"Education — Writing Assessment and AI Feedback Analytics","task_short_name":"kisspro_ai_feedback_revision_provenance_40_v1","operating_system":"Linux","task_description":"Objective: Assess the secondary students OWN performance in a writing class. Therefore, for every revised student essays, determine whether each the word had already been produced before AI feedback became available or first emerged during AI-feedback-supported revision.\n\nProfessional context: The revised essays come an authentic classroom-based writing-feedback experiment with secondary-school students. Students wrote an argumentative essay and then revised the same essay after receiving individualized GPT feedback on content, structure, and language style. The task models a teacher-facing product audit: it distinguishes writing performance that the student had already demonstrated independently from wording first produced during AI-supported revision.\n\nStarting state: The participant package contains 40 final revised essays with fixed word IDs and exact character offsets, a prediction template, task context, and 20 person-disjoint worked examples drawn from the same outcome distribution. Each worked example includes the pre-feedback draft, AI feedback, final revision, and correct word-level labels. Test-case drafts, feedback messages, process traces, and labels are sealed.\n\nInstructions and rules: For every supplied test word, output exactly one of two labels. PRE_AI_FEEDBACK_WRITING means the word is preserved from the student's pre-feedback draft. AI_FEEDBACK_PHASE_WRITING means the word first appears in the revision completed with AI feedback available. The second label is temporal and support-based; it does not claim that GPT literally authored the word. Include exactly the 40 supplied case IDs and every supplied word ID exactly once. Do not add cases, omit words, or alter tokenization.\n\nDeliverable: Complete the supplied prediction template and save the result as predictions.jsonl.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Run evaluator/score_predictions.py predictions.jsonl against the sealed gold.jsonl. The scorer requires exactly the 40 supplied unique case IDs and rejects missing, duplicate, additional, or unknown cases. Within every case, each supplied word ID must appear exactly once with one valid label. A document counts as correct only if every word label matches gold; the primary score is exact documents divided by 40. Secondary diagnostics are overall word accuracy, per-label F1, macro-F1, and exact transition-boundary F1. The evaluator package includes deterministic gold derivation, sealed process evidence, provenance records, strict controls, and a package validator. The perfect control must score 40/40; both all-pre-feedback and all-feedback-phase controls must score 0/40.","software_and_version":"Python 3.11+; UTF-8 JSONL; standard-library evaluator","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"a37b0aef-00f9-46cd-9da3-993ade268412","filename":"kisspro_ai_feedback_revision_provenance_40_v1_participant.zip","category":"input_materials","content_type":"application/zip","size_bytes":251677},{"file_id":"a7528ec3-2bc3-4487-b73f-c4a472c9c5c3","filename":"kisspro_ai_feedback_revision_provenance_40_v1_reference.zip","category":"reference_output","content_type":"application/zip","size_bytes":503125},{"file_id":"8c3b0330-7040-4a8f-9f86-508c9f087d19","filename":"kisspro_ai_feedback_revision_provenance_40_v1_gpt55_selftest.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":16441}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, objectively evaluable, and end-to-end reachable, with only a non-material checksum inconsistency in the optional self-test evidence.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, two temporal provenance labels, fixed tokenization, required 40 cases, and exact JSONL deliverable are clear.\n- Inputs: Complete — The participant package contains 40 test essays, 20 person-disjoint worked examples, fixed word IDs, verified character offsets, context, and a prediction template.\n- Toolchain: Complete — Python 3.11+, UTF-8 JSONL, and a standard-library evaluator provide a legally obtainable single-machine workflow.\n- Reference output: Complete — The sealed package contains gold labels, paired snapshots, feedback, process evidence, source provenance, controls, and deterministic derivation logic for all 40 cases.\n- Rubric: Complete — Exact-document accuracy is primary, with word accuracy, per-label F1, macro-F1, and transition-boundary F1 providing meaningful partial-credit diagnostics.\n\n**File and Task Reachability:**\nAttachment classification: reachable. The inspected files were `kisspro_ai_feedback_revision_provenance_40_v1_participant.zip`, `kisspro_ai_feedback_revision_provenance_40_v1_reference.zip`, and `kisspro_ai_feedback_revision_provenance_40_v1_gpt55_selftest.zip`. All three were readable, unencrypted ZIP archives containing regular relative files without traversal paths, links, devices, or excessive expansion. All JSON and JSONL records were parsed, all 25 internal manifest sizes and hashes matched, the 11,048 test-word mappings and offsets were consistent, the documented alignment reproduced all 40 gold documents exactly, and all 40 keylogs were structurally parseable. No attachment was left uninspected, although student prose and individual key events were schema-checked rather than manually read exhaustively. The supplied Python was statically inspected but could not be executed because Python was unavailable in the review environment; independent validation reproduced the perfect, single-label, and self-test metrics.\n\nThe self-test report names a participant-archive checksum different from the uploaded participant ZIP. Its predictions nevertheless match every current case and word ID, and its reported metrics reproduce exactly, so this does not block the task but weakens archive-level self-test provenance.\n\nTask classification: reachable. A solver can inspect the participant records, complete `predictions.jsonl`, and have it scored against the sealed gold with strict case, word-ID, and label validation on one ordinary machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative screening evidence was found; public identifiability could not be independently checked within the permitted evidence boundary.\n\n**Recommended Changes:**\nThe core benchmark, evaluator, controls, and provenance evidence are ready; no material changes are required. For audit hygiene, regenerate the self-test report against the exact submitted participant ZIP or correct its participant checksum.","reviewed_at":"2026-08-27T20:12:01.246027Z"},"claim":{"id":"5f297a6a-f5c1-4108-a752-85c6e2eae150","submission_id":"4fce97e8-3811-49b9-b859-a5d6d8c2a595","submission_revision_id":"5f6281cc-d5b5-4df5-9c15-b9d3b4d8a757","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:12:57.489373Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}