{"submission_id":"2d350ba9-f770-43b7-9579-faaf04ccd76d","submission_revision_id":"52e40aa9-c4e4-4f7a-af72-9ac88d26f464","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-28T14:34:03.329315Z","submitter_name":"Thorben Jansen","submitter_email":"tjansen@ipn.uni-kiel.de","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT-5.5","harness":"Codex multi-agent subagents (high reasoning)","estimated_score":"10–20%"}},"submitted_at":"2026-08-28T14:34:03.329315+00:00","industry_domain":"Educational Assessment / Learning Analytics / Writing-Process Research","task_short_name":"classroom_process_checkpoint_reconstruction_25","operating_system":"Linux","task_description":"Professional Context and Objective: When teachers evaluate writing, the final submitted text may not be sufficient. They may also need to evaluate the writing and revision process—for example, which ideas were already present at an earlier stage, how the text changed after feedback, and what the student actually had written at each pedagogically relevant moment. This requires reconstructing exact historical text states from the available usage and process data. Reliable agent support for this reconstruction is therefore a prerequisite for process-aware assessment at scale.\n\nFor 25 writing cases, reconstruct the exact complete student text at every recorded AI-feedback request and at final submission. The cases contain unpublished data collected in authentic classrooms as part of a scientific study. The reconstructed states must match the versions independently stored by the application.\n\nStarting State: The agent receives complete original raw JSONL process exports, including all text operations, keyboard and beforeinput activity, paste/cut events, clickstreams, sidebar activity, session metadata, and other recorded payloads. A manifest identifies the ordered checkpoint outputs but withholds both their timestamps and the independently stored evaluation texts. Two simple, person-distinct worked examples include raw logs and exact targets. No decoder, simplified events, schema explanation, marker rule, or event-semantics guide is supplied.\n\nInstructions and Rules: Infer the raw events that triggered the independently stored checkpoints and the correct reconstruction procedure from the raw records and worked examples. Produce every required UTF-8 .txt file without correcting or normalizing student language.\n\nExpected Deliverable: submission/ containing exactly 168 checkpoint files across 25 cases.\n\nSolvability: This is an authentic assessment-record reconstruction and data-quality workflow. Every hidden output has been certified as exactly derivable from the supplied raw logs by a frozen evaluator-side reference decoder. The task therefore measures agent reconstruction errors rather than unrecoverable targets.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Run python3 evaluation/validate_package.py --input-dir input --reference-dir reference_output, then python3 evaluation/score_submission.py --manifest input/evaluation_manifest.jsonl --gold-dir reference_output --submission-dir submission. The primary score is completely exact cases divided by 25; a case counts only when every requested checkpoint is byte-exact. Text-state accuracy and character similarity are diagnostics.","software_and_version":"Python 3.11+; UTF-8 JSONL and text-processing environment","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"0eaf3036-a3a0-47f6-9a1d-b1893a8cfa86","filename":"01_KisKIan_MarkerInference25_Raw_Process_Input_with_Training.zip","category":"input_materials","content_type":"application/zip","size_bytes":2293108},{"file_id":"6889ce24-0b21-42ec-9322-c5bc834e9313","filename":"02_KisKIan_MarkerInference25_Hidden_Gold_and_Evaluator.zip","category":"reference_output","content_type":"application/zip","size_bytes":367511},{"file_id":"dda80d95-6f32-4641-a486-4c231be1aa70","filename":"03_KisKIan_MarkerInference25_GPT55_Blind_SelfTest_Evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":64442}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally grounded, fully reachable, and supported by consistent inputs, hidden references, provenance, evaluator controls, and blind difficulty evidence.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly defines the 25-case reconstruction task, inference constraints, byte-exact UTF-8 output contract, and required 168-file directory structure.\n- Inputs: Complete — Contains 1,545 evaluation and 104 training raw JSONL files, two person-distinct worked examples, 25 evaluation cases, and 168 unique checkpoint specifications; all referenced files exist, parse correctly, and match their manifest hashes.\n- Toolchain: Complete — Python 3.11+ and standard UTF-8/JSON processing are sufficient; evaluator code uses the standard library and is compatible with the stated Linux workflow.\n- Reference output: Complete — Exactly 168 gold text files are present with matching manifest hashes, hidden checkpoint timestamps, a frozen replay decoder, certification provenance, and participant/training separation.\n- Rubric: Complete — The scorer implements the stated all-checkpoints-exact case metric, exact-state and character-similarity diagnostics, structural checks, and verified perfect and empty controls.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`. The inspected files were `01_KisKIan_MarkerInference25_Raw_Process_Input_with_Training.zip`, `02_KisKIan_MarkerInference25_Hidden_Gold_and_Evaluator.zip`, and `03_KisKIan_MarkerInference25_GPT55_Blind_SelfTest_Evidence.zip`. All three matched their manifest sizes, were readable unencrypted ZIP archives, passed integrity checks, contained no unsafe paths or special entries, and had modest expansion sizes. Every JSON/JSONL artifact was parsed and checked for valid UTF-8; representative raw records and schemas, all manifest relationships, raw and gold hashes, provenance summaries, evaluator logic, score controls, and blind-run reports were inspected. The self-test input hash matches the submitted input archive, and each run’s 168 submission hashes reproduces its reported exact-state count. Individual student text bodies and every line of exploratory self-test helper code were not manually reproduced; uploaded Python was not executed, but key evaluator sources were inspected statically as required for safe review. An ordinary solver can infer the checkpoint markers from the worked examples, reconstruct the requested files on one machine, and be evaluated objectively with the supplied validator and scorer. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports a screening label; external public-identifiability was not checked because network search is outside the permitted review boundary.\n\n**Recommended Changes:**\nThe submission provides unusually strong integrity, provenance, separation, solvability, and difficulty evidence. No changes are required.","reviewed_at":"2026-08-28T14:45:41.079243Z"},"claim":null}