{"claim":{"id":"18ca2868-4af4-4f56-a0f5-62c197c78500","submission_id":"bb21baed-b817-43c2-a3e6-67db48c945a1","submission_revision_id":"ff0d4e67-2ddc-4696-b369-a478c5c5c139","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:28.057565Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"bb21baed-b817-43c2-a3e6-67db48c945a1","submission_revision_id":"ff0d4e67-2ddc-4696-b369-a478c5c5c139","revision_version":4,"submission_status":"under_review","submitted_at":"2026-08-25T02:48:49.829994Z","submitter_name":"JICHENG WANG","submitter_email":"jconliner67@gmail.com","task":{"status":"submitted","version":4,"extra_fields":{},"submitted_at":"2026-08-25T02:48:49.829994+00:00","industry_domain":"AI Research / Machine Learning Interpretability","task_short_name":"othello_gpt_world_model_repro_1","operating_system":"Linux","task_description":"Objective: Reproduce, end to end, the two headline results of \"Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task\" (Li et al., ICLR 2023, arXiv:2210.13382, \"Othello-GPT\"): (1) the trained model's legal-move top-1 error rate, and (2) the linear-versus-nonlinear dissociation of board-state probes trained on its internal activations, which is the paper's central evidence for an emergent NONLINEAR world model.\n\nStarting State: You are given the paper, the authors' public repository (https://github.com/likenneth/othello_world, pinned commit f23bb5696cf30b93bd8af8a391ee33fc3aac417e), and one provided file, gen_games.py, a deterministic generator for the evaluation game sets. Nothing else is set up: no environment, no data, no checkpoint. Note that all four Google Drive links in the repository README are dead (404), so the released artifacts must be recovered elsewhere; a byte-identical public mirror of the synthetic-dataset checkpoint is at https://huggingface.co/sbentley/othello-world-ckpts/resolve/main/gpt_synthetic.ckpt\n\nInstructions & Rules:\n1. Clone the repository at the pinned commit and build a working Python environment for its data/ and mingpt/ modules. (Note: the `pgn` import comes from the `pgnparser` package.)\n2. Obtain gpt_synthetic.ckpt (8 layers, 8 heads, d_model 512, vocab 61, block size 59) and place it at ckpts/gpt_synthetic.ckpt.\n3. Generate the two fixed game sets by running the provided, unmodified gen_games.py from the repository root, producing game_sets/eval_games.pkl (1000 games, seed 20230211) and game_sets/probe_games.pkl (2000 games, seed 20230212). Record the SHA256 of both files.\n4. METRIC 1 - legal-move top-1 error rate. For every game in eval_games.pkl and every prefix of length 1..len-1, take the model's argmax next-token prediction (greedy, no sampling) and test its legality with the rules engine in data/othello.py (OthelloBoardState): a prediction is an error if it is the pad token or an illegal move. Report error_count / total_prefixes. Because the model is causal, one forward pass per game read at every position is mathematically identical to the per-prefix loop and is allowed.\n5. METRIC 2 - board-state probe accuracies at layer 6, following train_probe_othello.py:\n   - Extract activations after transformer block 6 (GPTforProbing(mconf, probe_layer=6)) with the model in EVAL MODE, for every position of every game in probe_games.pkl (positions 1..min(len,59)); ground truth is OthelloBoardState.get_gt(moves, \"get_state\"), the 3-way white/empty/black state of all 64 tiles after each move.\n   - Call mingpt.utils.set_seed(42), then split the pairs 80/20 into train/test with torch.utils.data.random_split.\n   - Train two probes on the train split, each for 32 epochs with the repository's probe TrainerConfig (batch_size 1024, lr 1e-3, betas (0.9,0.999), lr_decay True, warmup_tokens 5*len(train), final_tokens 32*len(train), weight_decay 0):\n       (a) linear    BatteryProbeClassification(probe_class=3, num_task=64)\n       (b) nonlinear BatteryProbeClassificationTwoLayer(probe_class=3, num_task=64, mid_dim=256)\n     Call set_seed(42) immediately before constructing and training each probe. (The README's example --mid_dim 64 --epo 16 is tuned for the authors' much larger corpus and underfits here; the required setting is mid_dim 256, 32 epochs.)\n   - Report each probe's best (max over epochs) test-split accuracy, i.e. max(test_acc_cont) from mingpt/probe_trainer.py, and keep the full per-epoch trace that probe_trainer.save_traces() writes.\n6. Do not modify gen_games.py, the seeds, the split ratio, the probe hyperparameters, or the probed layer.\n\nDeliverables - a submission directory containing exactly:\n- results.json with exactly these six keys: legal_move_total_prefixes, legal_move_error_count, legal_move_error_rate, probe_n_pairs, probe_test_acc_linear, probe_test_acc_nonlinear\n- dataset_checksums.json with eval_games_sha256 and probe_games_sha256 (lowercase hex) for the two files generated in step 3\n- traces_linear_layer6.txt and traces_nonlinear_tl256_layer6.txt, the save_traces() outputs of the two probe training runs\n- the script(s) the solver wrote\n\nCompute: CPU is sufficient (roughly 15-25 minutes); a GPU may be used. No API keys and no paid services; internet access is needed only to clone the repository and download the ~97 MB checkpoint.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Fully automated and deterministic, by the attached grade.py run against a submission directory. It prints a final line \"SCORE: <0-1>\". No human judgement and no LLM judging.\n\nIMPORTANT - grading is provenance-based. Reported numbers alone must not earn credit, because the paper's published values are public and could simply be copied. Two checks verify that the experiment was actually executed, and they act as GATES: if either fails, the submission scores 0.0 regardless of how plausible its numbers are.\n\nGate A - dataset provenance (weight 0.10). The submitted dataset_checksums.json must carry the exact SHA256 of the two seeded game sets produced by the unmodified gen_games.py. These checksums live only in the evaluator-side reference.json and are deliberately NOT disclosed in the solver-visible task text, so reporting them correctly requires having generated the data.\n\nGate B - training provenance (weight 0.10). The full save_traces() output of both probe runs is required, and it is checked for being a real optimisation trace rather than a plausible-looking one. Specifically:\n  (a) The test-accuracy trajectory must have the right length and stay finite and inside [0, 1], and must sit close to the reference trajectory (mean absolute deviation at most 0.03, no epoch off by more than 0.10).\n  (b) Closeness alone is NOT sufficient and is not treated as evidence. The linear probe's test accuracy spans only about 0.015 across its 32 epochs, so a constant line - or a straight interpolation between the first and last values - sits inside any useful deviation budget. The grader therefore also requires the submitted curve to MOVE like a real one: its spread must be comparable to the reference curve's, and it must correlate with the reference epoch by epoch. A flat or hand-drawn curve has the wrong variance and no correlation, and is rejected.\n  (c) The rest of the trace is required too and checked the same way: the train-loss, test-loss and train-accuracy curves, plus both 32x60 stratified per-position accuracy matrices, validated for shape, finiteness and range. Fabricating all of these consistently with each other is not practical.\n  (d) Each reported accuracy must equal the maximum of the submitted trajectory to within 1e-6, so a number cannot be typed in independently of the run that supposedly produced it.\n  (e) Non-finite values are rejected before any comparison, because every arithmetic comparison involving NaN is False and a trace of NaNs would otherwise slip past the deviation limits.\nVerified against attacks: constant trajectories, linearly interpolated trajectories, and NaN-filled trajectories that report the correct final values all score 0.0; the genuine reference run scores 1.0.\n\nStructural requirements, enforced before scoring: results.json must carry exactly the six required keys (missing or extra keys fail the submission), every value must be a finite number, and the solver's own script(s) (.py or .ipynb) must be present, since the deliverable is an inspectable reproduction rather than a bare results file.\n\nPast the gates, six weighted numeric checks: legal_move_total_prefixes exactly equal to the fixed count for the seeded eval set (0.10); legal_move_error_rate inside the reference band AND internally consistent with the reported counts to within 1e-6 (0.20); probe_n_pairs exactly equal to the fixed count for the seeded probe set (0.05); linear probe accuracy within its reference band (0.15); nonlinear probe accuracy within its reference band, whose upper edge is capped so that a train/test-leakage or fabricated near-perfect value is rejected rather than rewarded (0.20); and the paper's qualitative headline, nonlinear minus linear at least 0.15 (0.10).\n\nAll reference values, bands and reference trajectories live in the evaluator-side reference.json in the attached bundle, never in the solver-visible text. The bands are several times wider than the cross-platform variance we measured (the pipeline is seeded, and macOS/CPU and linux/A100 runs agreed to six decimals), while still rejecting every wrong-setup mode we identified: wrong checkpoint, wrong probe layer, the README's default probe hyperparameters, leaving dropout on during activation extraction, and training/evaluating on all pairs instead of the split.\n\nUsage: python3 grade.py <submission_dir>. Standard library only, runs in under a second.","software_and_version":"Python 3.11, PyTorch 2.x, othello_world research codebase (commit f23bb56), Git, Hugging Face CLI","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"32dbcce1-c1f4-44e7-8549-5e61444667e7","filename":"02_input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":4301},{"file_id":"b186047d-c17e-4bfb-9cc2-d88b6293e7fc","filename":"02_reference_output_and_grader.zip","category":"reference_output","content_type":"application/zip","size_bytes":135409}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, objectively gradable, and provides a credible end-to-end reproduction path with strong provenance controls.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, fixed seeds, model configuration, evaluation protocol, probe settings, deliverables, and prohibited deviations are explicit.\n- Inputs: Complete — The deterministic generator is supplied, while the paper, pinned repository revision, and replacement checkpoint are precisely identified.\n- Toolchain: Complete — The required Linux, Python, PyTorch, Git, dependency, compute, and network requirements are practical and sufficiently specified.\n- Reference output: Complete — The bundle contains reference metrics, checksums, implementation scripts, full 32-epoch traces, a training-log excerpt, and a complete example submission.\n- Rubric: Complete — The standard-library grader defines structural checks, weighted metrics, provenance gates, finite-value validation, trajectory checks, and a deterministic score contract.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Both manifest attachments were inspected: `02_input_materials.zip` is a valid, unencrypted 4,301-byte ZIP containing `task_statement_solver.md` and `gen_games.py`; `02_reference_output_and_grader.zip` is a valid, unencrypted 135,409-byte ZIP containing `reference.json`, `grade.py`, reference scripts, metrics, checksums, logs, traces, and duplicate submission-ready artifacts. All 17 reference-bundle members were inspected or byte-compared with their inspected duplicates. The JSON and trace files parse correctly; both traces contain six expected arrays, 32 epochs, and 32x60 stratified matrices. No archive traversal paths, links, encryption, corruption, unsafe entries, or unexplained executables were found. No attachment content remained unreviewed. Uploaded Python sources were reviewed statically and were not executed. Availability of the documented public repository and checkpoint endpoint was not independently revalidated under the evidence boundary.\n\nTask classification: `reachable`. A solver can obtain the pinned code and checkpoint, generate the seeded datasets, evaluate legal moves, extract layer-6 activations, train both probes on one suitable machine, assemble the exact deliverables, and receive an objective score. The supplied reference results, checksums, traces, and scripts are mutually consistent with that path. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The task explicitly reproduces results from a named public paper using a public repository and checkpoint. This is materially mitigated by evaluator-hidden seeded checksums, full optimization-trajectory gates, internal metric consistency checks, and separation of solver-visible inputs from reference outputs, so copying published headline values cannot earn credit.\n\n**Recommended Changes:**\nNo changes are required.","reviewed_at":"2026-08-25T03:31:20.165020Z"},"claim":{"id":"18ca2868-4af4-4f56-a0f5-62c197c78500","submission_id":"bb21baed-b817-43c2-a3e6-67db48c945a1","submission_revision_id":"ff0d4e67-2ddc-4696-b369-a478c5c5c139","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:28.057565Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}