{"submission_id":"826f7805-0067-44c3-848e-b9a5fad514b1","submission_revision_id":"29599bdc-9377-408c-aabf-36c618dd9a6e","revision_version":2,"submission_status":"under_review","submitted_at":"2026-08-14T21:45:32.350829Z","submitter_name":"Yanxin Zhang","submitter_email":"yzhang2879@wisc.edu","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"`claude-sonnet-4-6`  `gpt-5.5`","harness":"agent","estimated_score":"10–20%"}},"submitted_at":"2026-08-14T21:45:32.350829+00:00","industry_domain":"AI Engineering & Systems — LLM inference serving, KV-cache memory management","task_short_name":"paged_kv_block_manager_instance_1","operating_system":"Linux","task_description":"Objective: Reconstruct a paged KV-cache allocator from recordings of its behaviour. There is no specification.\n\nStarting State: The scenario is that an LLM serving engine's KV-cache allocator has been deployed and its source lost. What survives is the scheduler above it, which is not the agent's to change, and the event logs that scheduler recorded while driving the allocator through serving traces.\n\nTo be precise about provenance: the traces and their event logs are synthetic. They are generated from a fixed seed by the generator shipped in the reference bundle, and replayed through a reference implementation to produce the expected logs. Nothing here is captured from a real deployment or derived from any third-party source. The framing above is a scenario, not a claim about where the data came from.\n\nThe agent is given twelve recorded traces, the exact event log the reference allocator produced for each one, and the replay harness the grader itself uses.\n\nInstructions & Rules: Write output/block_manager.py so that it reproduces the reference allocator's behaviour exactly.\n\nNo policy specification is supplied. Recovering the rules from the logs IS the task. What has to be worked out includes: which physical block is taken when several are free; which is evicted when none are, and on what ordering; what counts as \"using\" a block for that ordering (does reusing one from the cache count? does forking?); where a block goes when its reference count reaches zero, and whether it keeps its contents; what num_free_blocks() actually counts; what an append does when the tail block is shared, and whether a FULL shared tail still needs copying when the append is starting a new block; the exact watermark boundaries behind the OK / LATER / NEVER verdicts; which sequence preempt() evicts; and the three-way split between blocks a preemption released, blocks a fork sibling still holds, and blocks returned to the prefix cache.\n\nOnly what cannot be recovered from behaviour at all is stated outright: the chained SHA-256 hash function, the reference-counting arithmetic, and the error contract.\n\nDeliverables:\n- output/block_manager.py    the implementation\n- output/trace_results.json  its event log for the twelve examples, produced with the shipped replay harness\n- output/notes.md            which log evidence pinned each rule, and anything left uncertain\n","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The submitted implementation is replayed against 145 hidden traces (4,112 events) the agent has never seen, and the event log is compared field by field: physical block ids, cache-hit positions, copy-on-write pairs, what each preemption released versus kept shared versus returned to the cache, and the free-block count after every operation.\n\nHARD GATES:\n- any deliverable missing\n- block_manager.py does not define BlockManager, block_hash, or BlockManagerError\n- a replay raises before completing\n- trace_results.json disagrees with what the submitted implementation actually produces\n- ANY ONE of the 145 hidden traces fails to replay exactly\n\nThe reported score is the fraction of hidden events reproduced, for diagnosis only; passing requires every hidden trace to be exact. That distinction is not cosmetic: in measurement, every failing run still scored above 0.95 on events. An allocator that is right ninety-nine times in a hundred corrupts the cache for whichever request lands in the hundredth.\n\nThe twelve example traces are not scored - their expected logs ship to the agent, so reproducing them proves nothing.\n\nWHY THE REFERENCE IS EXACT: the reference allocator is deterministic and every policy is pinned, so each trace admits exactly one legal event log, and the reference log is generated by running that implementation rather than hand-written.\n\nWHY THE TASK IS FAIR WITHOUT A SPECIFICATION: the examples were not chosen by hand. Twenty candidate rules - every touch-stamp site, ordering choice, boundary comparison and copy-on-write trigger in the reference - were each perturbed to their plausible wrong alternative, and traces were selected by minimum set cover so that the shipped examples discriminate every rule the hidden set enforces. Two touch sites proved unobservable by any trace at all; those are not graded, so they cannot trap anyone.\n\nThe evaluator honours the grader's pass/fail verdict rather than its weighted score, so a submission that misses even one hidden trace scores zero regardless of how close the event fraction came.\n\nGRADER CALIBRATION - the reference implementation, and each single wrong inference:\n  the reference allocator                           145/145 exact   1.0000  pass\n  preempt the oldest sequence instead of the newest   34/145        0.7249  fail\n  report every block as released                      51/145        0.9607  fail\n  MRU eviction instead of LRU                        113/145        0.9579  fail\n  watermark boundary off by one                      104/145        0.9276  fail\n  copy a full shared tail when starting a new block  120/145        0.9800  fail\n  a cache hit does not refresh its touch stamp       139/145        0.9917  fail\n  fork does not stamp admission order                143/145        0.9988  fail\n","software_and_version":"Python 3.12, standard library only. Semantics follow vLLM-style paged attention: block tables, prefix caching, copy-on-write, LRU eviction, watermark admission, and recomputation-style preemption.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"f93d1447-7103-4a1c-a78a-6fd528b9b284","filename":"A2_paged_kv_block_manager_INPUT.zip","category":"input_materials","content_type":"application/zip","size_bytes":15156},{"file_id":"5589c476-88d6-4293-bee5-3a51dbb128d6","filename":"A2_paged_kv_block_manager_REFERENCE.zip","category":"reference_output","content_type":"application/zip","size_bytes":151268},{"file_id":"7fe7551f-8afc-43d3-8470-e215e7fecc52","filename":"paged_kv_block_manager_selftest.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":2936}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, professionally challenging, objectively graded, and reachable with only minor ancillary metadata inconsistencies.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, required API, explicitly supplied rules, inference scope, error contract, and three deliverables are clear.\n- Inputs: Complete — Twelve valid traces, 306 visible expected events, a replay harness, and a detailed brief are supplied.\n- Toolchain: Complete — Linux and freely obtainable Python 3.12 with the standard library are sufficient; no external services or packages are needed.\n- Reference output: Complete — The evaluator bundle contains the reference implementation, 145 matching hidden traces, 4,112 expected events, and the grader.\n- Rubric: Complete — Deliverable gates, compared fields, diagnostic scoring, and the requirement that every hidden trace match exactly are explicit and implemented.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. All three manifest attachments were inspected: `A2_paged_kv_block_manager_INPUT.zip` (15,156 bytes, 15 members), `A2_paged_kv_block_manager_REFERENCE.zip` (151,268 bytes, 154 members), and `paged_kv_block_manager_selftest.txt` (2,936-byte ASCII text). Manifest byte counts matched. Both ZIP archives were readable, unencrypted, CRC-valid, modest in expansion, and contained only regular files with no unsafe paths, links, or device entries. Every JSON member parsed, all trace files passed structural checks, and trace names matched their expected-log keys. The Python and Markdown materials were statically inspected without executing submitted code; individual hidden trace bodies were validated collectively and sampled rather than manually read line by line. The review environment lacked a Python interpreter for an additional non-executing syntax parse, which is a reviewer-tooling limitation rather than a submission defect.\n\nSolver-visible materials are separated from evaluator-only reference code, hidden traces, and hidden expected logs. A solver can infer the policies, implement the three outputs, generate the visible log with the supplied harness, and be evaluated by exact replay against the hidden set. The self-test evidence reports four completed trials, all producing the deliverables and visible events but failing hidden exactness, which supports the intended difficulty. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe core task, evidence split, and grader are ready; no material changes are required. As optional metadata cleanup, align ancillary `main.py` and `task_card.json` wording with revision 2 by consistently describing the traces as synthetic, correcting the stale “five traces” reference to twelve, and using 145-trace calibration denominators throughout.","reviewed_at":"2026-08-14T21:53:24.984906Z"},"claim":null}