{"submission_id":"9c3fa331-9321-4ccc-a136-e52f0c227380","submission_revision_id":"5658b58b-d2fd-4858-b8c9-48dfc0100056","revision_version":9,"submission_status":"submitted","submitted_at":"2026-08-21T10:59:48.616253Z","submitter_name":"Satyam Mohla","submitter_email":"satyam.mohla@dtc.ox.ac.uk","task":{"status":"submitted","version":9,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"<1%"}},"submitted_at":"2026-08-21T10:59:48.616253+00:00","industry_domain":"Compilers / Program Analysis (Engineering — Software)","task_short_name":"dataflow_fixpoint_instance_1","operating_system":"Linux","task_description":"Objective\nAct as the dataflow pass of an optimizing compiler. Given a control-flow graph (CFG) of three-address-code basic blocks, compute the four canonical bit-vector dataflow analyses to a FIXPOINT and report the IN and OUT set at every basic block:\n  - live  : live variables        (backward, may / UNION,        sets of variables)\n  - reach : reaching definitions  (forward,  may / UNION,        sets of definition ids)\n  - avail : available expressions (forward,  must / INTERSECTION, sets of expression ids)\n  - vbusy : very-busy expressions (backward, must / INTERSECTION, sets of expression ids)\n\nStarting State\n- /input/program.json -- a single self-describing file with two parts:\n  * \"program\": the CFG. \"blocks\" (each with an ordered list of three-address statements and a \"succ\" successor list), the \"entry\" and \"exit\" block ids, the variable list, a \"def_table\" (every definition statement has a unique id), and an \"expr_table\" (every distinct expression has a unique id; each binary statement carries its \"e\" expr-id and \"d\" def-id).\n  * \"task\": the four analyses with, for each, its direction (forward/backward), meet operator (union/intersection), element type, transfer function, local gen rule, boundary condition, interior initialisation, and universe -- plus the exact gen/kill and expression-identity rules.\n\nInstructions & Rules\n1. Iterate each analysis until no IN/OUT set changes (a FIXPOINT). A single forward/backward sweep is wrong on any CFG with a loop.\n2. Use the correct direction and meet for each analysis. may analyses (live, reach) meet with UNION and initialise interior sets to the EMPTY set; must analyses (avail, vbusy) meet with INTERSECTION and initialise interior sets to the FULL universe (top). The forward boundary (entry IN) and backward boundary (exit OUT) are empty.\n3. Compute the KILL sets precisely: reaching definitions -- a definition of v kills ALL program-wide definitions of v; gen is the downward-exposed definition of each variable in the block. available and very-busy share one kill rule -- a definition of v kills every expression that uses v as an operand.\n4. Get the two expression GEN sets right -- they are MIRROR IMAGES, not the same rule, and this is the single easiest thing to get wrong by analogy:\n   - avail egen[B] is DOWNWARD-exposed: an expression computed in B counts only if NO LATER statement in B redefines one of its operands (so `a = a + b` does NOT leave `a + b` available after the block).\n   - vbusy euse[B] is UPWARD-exposed: an expression computed in B counts only if NO EARLIER statement in B redefines one of its operands. Very-busy runs BACKWARD, so it asks what is anticipated on ENTRY to the block.\n   Each analysis in program.json carries its own \"local_gen\" field stating its exposure; that field is authoritative.\n5. Expression identity is the exact (op, arg1, arg2) tuple (NOT commutative); every binary statement carries its expr-id, and the expr_table is authoritative. Use the published ids as set elements (variable names for live; def-ids for reach; expr-ids for avail/vbusy).\n6. Write /output/dataflow.json (see Output Specification below).\n\nOutput Specification\nWrite exactly one file: /output/dataflow.json = {\"live\": {\"<block>\": {\"in\":[...], \"out\":[...]}, ...}, \"reach\": {...}, \"avail\": {...}, \"vbusy\": {...}}. Emit all four analyses, and for each, the IN and OUT set of EVERY block (keyed by block id). Sets are lists of the published ids; order does not matter and duplicates are ignored.\n\nAllowed tools: Python 3 + NumPy (open-source). Professional equivalent: the dataflow framework of an optimizing compiler (LLVM/GCC analysis passes) or a standalone static-analysis tool. The deliverable is, in effect, a correct dataflow solver for the given program.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Grading is deterministic set equality -- no human or LLM judge, no floating-point tolerance, so it is byte-reproducible. For every analysis, every block, and each slot (IN, OUT), the agent's set of ids must exactly equal the grader's. Because a monotone dataflow framework over a finite lattice has a mathematically UNIQUE maximal-fixpoint solution (Kildall), the grader regenerates this instance's CFG from the seed and recomputes the answer with its own solver -- so for the live/private seed there is no stored answer and nothing to leak. The COMPLETE evaluation harness is bundled in the uploaded reference/evaluation zip (the grader/generator _shared_eval.py, the ALE hooks main.py, an independent reference solver, selfcheck.py, a worked public example, and the submitted instance with its reference output); nothing is provided separately, and the platform needs only Python 3 (plus NumPy on the evaluator side). At runtime the agent receives ONLY /input/program.json.\n\nGate-and-score procedure:\n1. Gates (score 0 on a non-object submission; otherwise missing pieces simply fail their sets): /output/dataflow.json present and a parseable JSON object; each analysis present; each block present with in/out lists.\n2. Set equality: for every analysis, block, and slot in {in, out}, compare the agent's set against the grader's (regenerated from the seed) for EXACT equality.\n3. Score = fraction of matching sets; full credit (1.0) only if EVERY set matches.\n4. Pass/Fail cutoff: Pass = 1.0; any score below 1.0 is a Fail. The continuous score is also returned for diagnostics, and the cutoff is a single exposed constant (PASS_THRESHOLD, default 1.0) so a softer bar is a one-line change. Rationale, the difficulty profile, and calibration are documented in docs/DIFFICULTY_PROFILE.md (reproducible via selfcheck.py and evaluations/agent_baselines.py): only a fully-correct solver reaches 1.0; competent single-rule slips score high but 0/6. The published specification and the grader are held together by an explicit check: selfcheck.py check [7] derives behaviour from the emitted agent-visible spec, confronts it with a hand-built block where upward and downward expression exposure disagree, and confirms a solver that conflates the two exposure rules is caught (0.856 mean, 0/6). A solver can therefore implement exactly what /input/program.json says and reach 1.0.\n\nIMPORTANT -- the private evaluation path is REGISTERED, not just recommended. main.py registers two splits: load() is the PUBLIC split (seeds 1-6, whose answers ship in the package, for development and dry-runs only) and load_heldout() is the PRIVATE split (held-out seeds outside 1-6, for which no answer exists anywhere in the package). Score the private split. The infrastructure REGENERATES each instance in-process from its seed (start() calls generate_program(seed)); it never reads the uploaded program.json, so any held-out seed yields a fresh, previously unseen CFG and the grader recomputes the unique fixpoint itself. The bundled local harness (_local_harness/run_local.py) exercises BOTH splits end to end and reports 1.000 for the reference solver and 0.000 for a missing output on every seed of each. RECORD the chosen (seed, mode) with the run: generation is fully deterministic, so logging that pair reproduces the exact CFG and its ground-truth fixpoint byte-for-byte on any machine (see INTEGRATION.md \"Reproducibility\").","software_and_version":"Python 3. The AGENT needs only plain Python 3 (standard library) -- the problem is solvable with ordinary set operations, no third-party packages. NumPy is used ONLY by the bundled evaluation code (the CFG generator's seeded RNG and the golden solver/grader), not by the agent's solution; it is the sole third-party dependency (requirements.txt included). Everything is open-source, headless, self-contained. No additional requirements: no special hardware, GPU, network access, credentials/accounts, or setup steps -- a stock Linux + Python 3 environment suffices. Professional equivalent: the dataflow framework of an optimizing compiler (LLVM analysis passes, GCC) or a standalone static-analysis / program-verification tool.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"d50a75ec-a2b9-4b72-a982-fecb32e374bb","filename":"dataflow_fixpoint_input.zip","category":"input_materials","content_type":"application/zip","size_bytes":2856},{"file_id":"27dfb281-26dd-4647-9f5e-c172029c9d16","filename":"dataflow_fixpoint_reference_and_evaluation(2).zip","category":"reference_output","content_type":"application/zip","size_bytes":66441}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally scoped, safely reachable, and supported by a coherent private evaluation path with independently verified reference results.\n\n**Completeness Check:**\n- Instruction: Complete — The four analyses, directions, meet operators, initialization, boundaries, transfer rules, expression identity, fixpoint requirement, and output schema are explicit.\n- Inputs: Complete — The solver input is valid JSON containing a nine-block CFG, six variables, 32 definitions, 16 expressions, authoritative tables, and analysis-specific rules.\n- Toolchain: Complete — Python 3 and NumPy are open-source, headless, single-machine compatible, and require no external services or specialized hardware.\n- Reference output: Complete — Reference outputs and an independent worklist solver are supplied. Independent recomputation confirmed all 72 submitted-instance IN/OUT sets.\n- Rubric: Complete — Deterministic set equality covers every analysis, block, and IN/OUT slot, provides a continuous score, and requires 1.0 to pass.\n\n**File and Task Reachability:**\nThe attachment classification is `reachable`, and the end-to-end task classification is `reachable`. Both ZIP files matched their declared sizes, passed integrity checks, had modest expansion, and contained no encryption, traversal paths, links, device entries, or opaque formats.\n\nFiles inspected were `program.json`; `main.py`; `_shared_eval.py`; `reference_solution.py`; `selfcheck.py`; `requirements.txt`; `README.md`; `SUBMISSION_FORM.md`; `task_card.json`; `INTEGRATION.md`; `docs/FAQ.md`; `docs/DIFFICULTY_PROFILE.md`; `docs/PROVENANCE.md`; `_local_harness/run_local.py`; `evaluations/agent_baselines.py`; and the example and submitted-instance input, output, and grade-report JSON files. All JSON parsed successfully, and the submitted input matched the packaged submitted-instance input. The long FAQ was reviewed structurally and through relevant passages; no binary or opaque content remained unreviewed.\n\nThe private split is statically registered, uses seeds disjoint from published and submitted examples, generates the solver-visible input in process, and regenerates the same program for evaluation without staging evaluator assets. The review runtime lacked Python 3, so bundled Python programs were not executed; source was inspected statically, and a separate read-only implementation verified CFG integrity, table cross-references, reachability, schema, vocabularies, and the reference fixpoint. The unconfirmed live `cua_bench` symbol names are routine evaluator wiring, not a material task gap.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone — the submission describes clean-room, procedurally generated inputs rather than AI-authored evidence or a predetermined defect fixture, and no scored answer is exposed. Public identifiability was not externally searched because review was limited to submitted evidence; nothing inspected affirmatively supports a public-benchmark label.\n\n**Recommended Changes:**\nThe core task and evaluation package are ready; no material changes are required. Optional housekeeping is to correct the standalone reference solver’s stale no-argument default input filename and replace claims that all public-seed answers ship with the accurate statement that one public example answer ships.","reviewed_at":"2026-08-21T11:06:46.348701Z"},"claim":null}