{"submission_id":"dd740e81-c0a4-4123-95aa-68fb05882aef","submission_revision_id":"498a87e2-8004-4842-9a64-f140fee54936","revision_version":2,"submission_status":"under_review","submitted_at":"2026-08-16T08:08:59.650319Z","submitter_name":"Yanxin Zhang","submitter_email":"yzhang2879@wisc.edu","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"claude-sonnet-4-6 and gpt-5.5","harness":"agent","estimated_score":"10–20%"}},"submitted_at":"2026-08-16T08:08:59.650319+00:00","industry_domain":"Computer & Electronics Engineering - CPU microarchitecture, performance modelling and instruction-table construction","task_short_name":"uarch_timing_model_recovery_instance_1","operating_system":"Linux","task_description":"Objective: Recover the timing model of a small, undocumented CPU core from a log of timing measurements, and deliver it as code that predicts cycle counts exactly.\n\nStarting State: The scenario is that a core was characterised once and then went away. What survives is a log: 60 instruction sequences and, for each, the number of cycles the core took to run it starting from idle.\n\nThe agent cannot run the machine. There are no new experiments, no probing an opcode in isolation, no extra measurements. Sixty sequences is what there is, and they are ordinary mixed code rather than micro-benchmarks - nobody sat down and wrote one clean experiment per parameter. They vary in length, in how heavily their instructions depend on one another, and in which opcodes appear at all, and their names carry no information. Everything about the machine is in there jointly, and separating it is the work.\n\nTo be precise about provenance: the machine is invented for this task and the measurements are synthetic. Sequences are generated from a fixed seed by the build script shipped in the reference bundle, and their cycle counts come from running the reference model. This is not a model of any real processor and nothing derives from any third-party or proprietary source. The framing is a scenario, not a claim about where the data came from.\n\nInstructions & Rules: Write output/model.py defining predict_cycles(sequence) -> int, standard library only, run directly by the grader and not permitted to read from disk.\n\nNo specification of the machine is supplied. Recovering it from the log IS the task, and what has to be recovered is rules as well as numbers: how many instructions may start in one cycle; for each opcode, how long its result takes to become available; for each opcode, how long it ties up the port it runs on, which is a different number and for one opcode differs sharply from it; which opcodes contend for the same port; the order the machine is allowed to start instructions in relative to program order; which port an opcode takes when it could use more than one; what event the reported cycle count is measured to; and what it costs to forward a result from one instruction to the next, which is not the same for every pair of opcodes.\n\nThose last four are rules rather than integers, and each has a plausible alternative the log can distinguish. They are also the most expensive things to get wrong: missing the cross-domain forwarding cost entirely leaves only 16 of the 180 graded sequences correct.\n\nOnly facts that no measurement could reveal are stated outright: that timing is deterministic, that there is no cache or memory hierarchy so loads never miss, that every measurement starts from an idle machine, and that the model is small.\n\nDeliverables:\n- output/model.py   the predictor\n- output/notes.md   what was concluded about the machine and which measurements forced each conclusion\n","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The submitted predict_cycles is run over 180 instruction sequences the agent has never seen, drawn the same way as the 60 it was given, and its integer cycle counts are compared against the machine's. Nothing the agent wrote in prose is read or scored; only what its code predicts.\n\nHARD GATES:\n- either deliverable missing\n- model.py does not define predict_cycles, or fails to import\n- predict_cycles raises, or returns a non-integer\n- ANY ONE of the 180 hidden sequences is not predicted exactly\n\nThe reported score is the fraction of hidden sequences predicted exactly, for diagnosis only; passing requires all 180. That distinction is load-bearing rather than decorative: a model whose only error is an issue width of 4 instead of 3 scores 0.9167, which clears any reasonable score threshold while being, as a timing model, simply wrong. The 60 visible measurements are not scored - their answers ship to the agent - but are checked so that a submission contradicting what it was shown is reported as such.\n\nEVALUATOR ISOLATION: the submission runs in a separate interpreter that is handed the module path and two scratch files and never learns where the reference lives, so there are no grader frames to walk and no reference path in its argv. For the duration of that run the reference is held in the grader's memory and removed from the filesystem entirely, then written back - relocating it is not enough, because on a single-user filesystem there is no directory a same-user process cannot read. An adversarial test that sweeps the filesystem, reads sys.argv and the environment, and walks caller frames scores 0.0.\n\nWHY THE REFERENCE IS EXACT: the machine is deterministic, so each sequence has exactly one correct cycle count, and the expected counts are generated by running the reference model rather than written by hand. Comparison is integer equality.\n\nWHY THE TASK IS FAIR WITHOUT A SPECIFICATION: two independent checks, and the second caught what the first missed. First, every parameter and every rule was mutated to its plausible wrong alternative - 55 mutations covering issue width, per-opcode latency, occupancy and port assignment, the three discipline choices, the forwarding penalty and every opcode's forwarding domain - and the build refuses to emit data unless the log separates each mutation the hidden set punishes from the truth. That proved insufficient on its own, so second, each parameter is swept exhaustively: 166,659 alternative values, confirming that every one reproducing all 60 measurements also predicts all 180 hidden sequences correctly. Two earlier drafts failed that sweep and were fixed by adding measurements, never by narrowing what is graded.\n\nA DESIGN NOTE WORTH HAVING: an earlier version of this task shipped a textbook probe suite - a latency chain and throughput batches per opcode, all 28 opcode pairings, port-pressure tests, each experiment named for the parameter it isolated. It passed the same fairness checks, and an anchor model solved it in ten turns and three minutes at a perfect score, reading each parameter straight off the probe built to expose it. The information has to be present; it does not have to arrive one parameter per labelled experiment. The visible set is now 60 unlabelled mixed sequences, active experimentation is ruled out, and the same anchor reached 25 of 180 in 45 minutes.\n\nGRADER CALIBRATION - the real machine, and each single wrong inference:\n  the real machine                                180/180  1.0000  pass\n  forwarding penalty 1 -> 2                        13/180  0.0722  fail\n  cross-domain forwarding penalty missed entirely  16/180  0.0889  fail\n  issue out-of-order rather than in-order          40/180  0.2222  fail\n  div occupancy 12 -> 13                           96/180  0.5333  fail\n  count ends at last instruction, not last result  97/180  0.5389  fail\n  load latency 5 -> 4                              99/180  0.5500  fail\n  issue width 3 -> 2                              107/180  0.5944  fail\n  shuf latency 2 -> 3                             117/180  0.6500  fail\n  load moves to the integer domain                118/180  0.6556  fail\n  port choice lowest-numbered, not earliest-free  121/180  0.6722  fail\n  fma loses port 1                                160/180  0.8889  fail\n  issue width 3 -> 4                              165/180  0.9167  fail\n","software_and_version":"Python 3.12, standard library only. No third-party dependencies and no simulator to install: the deliverable is a self-contained predictor that the grader runs directly.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"2206c5ac-8e8b-45e7-8bf2-5fea9422b037","filename":"B2_uarch_timing_model_recovery_INPUT.zip","category":"input_materials","content_type":"application/zip","size_bytes":8236},{"file_id":"ba0160f9-ffb8-41e5-8cdf-8395b7634eea","filename":"uarch_timing_model_recovery_selftest.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":7266},{"file_id":"9530ba6e-37b2-4419-b9a0-c1a1b3569486","filename":"B2_uarch_timing_model_recovery_REFERENCE.zip","category":"reference_output","content_type":"application/zip","size_bytes":41884}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, reachable, and objectively gradable; inconsistent version-specific diagnostics should be synchronized but do not block execution or evaluation.\n\n**Completeness Check:**\n\n- Instruction: Complete — The objective, instruction schema, constraints, deliverables, and exact-cycle requirement are explicit.\n- Inputs: Complete — The brief, 60 labeled measurements, and eight-opcode list are readable, well-formed, and sufficient for the stated workflow.\n- Toolchain: Complete — Linux and Python 3.12 with the standard library provide a practical single-machine workflow without external services or specialized hardware.\n- Reference output: Complete — The bundle supplies the reference model, 180 hidden sequences, matching cycle labels, grader, isolation helper, generation script, and identifiability checker.\n- Rubric: Complete — Import, deliverable, return-type, runtime, and exact-match gates are deterministic; passing requires 180/180 hidden predictions.\n\n**File and Task Reachability:**\n\nAttachment classification: `reachable`. Task classification: `reachable`.\n\nAll manifested files were inspected: `B2_uarch_timing_model_recovery_INPUT.zip`, `uarch_timing_model_recovery_selftest.txt`, and `B2_uarch_timing_model_recovery_REFERENCE.zip`. Both ZIPs match their declared format and size, pass container-integrity checks, are unencrypted, and contain only bounded regular files with safe relative paths. All 14 archive members were reviewed, including Markdown, JSON, and static Python source; there was no unreviewed content or material tooling limitation.\n\nThe input contains 60 unique sequences of 6–40 instructions, valid opcode references, integer cycle counts, and no overlap with the 180 unique hidden sequences. Hidden sequence and cycle keysets match. Independent static-model validation reproduced all 60 visible and 180 hidden labels. The completion path is clear: infer the model, produce `model.py` and `notes.md`, then run the predictor in isolation and compare every hidden result by integer equality.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\nNone\n\n**Recommended Changes:**\n\nThe core task, data, and evaluator are valid. No changes are required for reachability or grading. Before publication, synchronize the submission narrative and self-test with the bundled revision: their mutation scores and anchor result differ from the current hidden set and task card. Also revise the brief’s claim of proven global identifiability to match the checker’s actual evidence: exhaustive bounded single-opcode sweeps plus randomized multi-parameter search, not an exhaustive proof over the full joint model space.","reviewed_at":"2026-08-16T08:14:38.498731Z"},"claim":null}