{"submission_id":"8421e69a-8f34-4d18-892c-cbad2f5e922e","submission_revision_id":"ddabe7b1-facd-4a76-a5f8-95b69ee34ec3","revision_version":4,"submission_status":"submitted","submitted_at":"2026-08-20T13:21:38.980969Z","submitter_name":"Sidharth Mohla","submitter_email":"sidharthmohla@gmail.com","task":{"status":"submitted","version":4,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5","harness":"Claude Code","estimated_score":"<1%"}},"submitted_at":"2026-08-20T13:21:38.980969+00:00","industry_domain":"Computer Architecture / Hardware Verification (Engineering — Systems)","task_short_name":"rx16_isa_conformance_instance_1","operating_system":"Linux","task_description":"Objective\nDo pre-silicon bring-up of a small 16-bit RISC core, \"RX16\": implement the golden instruction-set simulator from the written ISA specification and pass the ISA-conformance suite. For each test program, execute it from the given initial state to quiescence and report the final architectural state. The ISA is CUSTOMIZED per test case -- opcode numbering, register-field and condition-code encodings, immediate widths and sign-extension, and two semantic options (the PC-relative branch base; the multi-register load/store direction + write-back) are all mutated per seed -- so you must read THIS spec and implement it exactly, not recall any public or textbook ISA.\n\nStarting State\n- /input/isa_spec.json -- the authoritative, self-describing spec of this instance's RX16 ISA: the instruction set with per-instruction flag/write behavior, the exact 16-bit field layout, the (mutated) opcode/register/condition encodings, immediate widths + sign-extension, the predication rule, the flag definitions, and the semantic toggles.\n- /input/programs.json -- the conformance suite: a list of programs, each with \"words\" (assembled 16-bit instruction words) and its COMPLETE initial architectural state: \"regs0\" (initial R0..R7), \"mem0\" (initial 256-word data memory) and \"flags0\" (the initial Z/N/C/V condition flags). The initial flags are published, not assumed -- a predicated instruction may be reached before anything sets a flag.\nThis instance is one of an unlimited seeded family; every seed describes a genuinely different machine.\n\nInstructions & Rules\n1. Parse isa_spec.json and build a decoder/executor for exactly this ISA. Implement EVERY instruction the spec defines -- the suite exercises all of them, including the absolute register jump.\n2. Implement each instruction's semantics precisely, including: the flag-update rules and how they interact with predication (does a skipped predicated op still update flags?), the PC-relative branch base, the per-format immediate sign-extension widths, and the exact multi-register load/store address-update direction and write-back.\n3. For each program, load its full published initial state -- regs0, mem0 AND flags0 -- and run to quiescence. QUIESCENCE / HALT (also defined in isa_spec.json -> fields \"execution_model\" and \"max_steps\"): the program stops at the HLT instruction, OR when the PC leaves the code range [0, 2*code_len) (PC is a byte address; each 16-bit instruction is 2 bytes), OR after max_steps instructions -- whichever comes first.\n4. Write /output/results.json (see Output Specification below).\n\nOutput Specification\nWrite exactly one file: /output/results.json = {\"results\": [ {\"regs\": [8 ints R0..R7], \"flags\": {\"Z\":0/1,\"N\":0/1,\"C\":0/1,\"V\":0/1}, \"mem_window\": [64 ints]}, ... ]}. Emit EXACTLY one entry per program, in the SAME order as the input suite (results[i] corresponds to programs[i]) -- no missing entries and no extras. \"regs\" is the 8 final registers R0..R7; \"flags\" is the four final condition bits; \"mem_window\" is the first 64 words of data memory. Every value must be a genuine JSON integer: registers and memory words in 0..65535, flags exactly 0 or 1. Strings, floating-point values and out-of-range numbers are rejected by a structural gate and score 0 -- nothing is coerced or masked into range.\n\nAllowed tools: Python 3 + NumPy (open-source). Professional equivalent: a SystemVerilog RTL core co-simulated against a Sail/Spike-style golden ISA model with riscv-dv-style conformance testing. The deliverable is, in effect, an instruction-accurate simulator for the given ISA -- a genuine verification artifact.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Grading is deterministic differential testing -- no human/LLM judge. Comparison is bit-exact integer equality of the final architectural state (no floating-point tolerance), so it is byte-reproducible across machines. The grader regenerates this instance's ISA and the golden final state from the seed using its own simulator, and the per-seed ISA mutation makes every public CPU implementation a wrong answer, so the task is contamination-proof and there is no answer file to leak. At runtime the agent receives ONLY /input/isa_spec.json and /input/programs.json; the golden simulator, grader, reference, seed, and golden outputs are evaluator-only and are never staged to the agent (public_spec() strips golden state). The ISA generator, golden simulator, and grader are all in the same bundled file _shared_eval.py (included). An independent reference simulator and a passing example are also included (examples/seed1/grade_report.json shows 25/25, overall 1.0). Requires only Python 3 + NumPy.\n\nApply this gate-and-score procedure:\n1. Gates (score 0 on any failure, with no partial credit): /output/results.json present and parseable; \"results\" is a list whose length EXACTLY equals the program count (extra entries fail just as missing ones do); each entry is an object with exactly \"regs\" (8 values), \"flags\" (exactly Z/N/C/V) and \"mem_window\" (64 values); every value is a genuine JSON integer -- strings, floats and booleans are rejected rather than converted; registers and memory words lie within 0..65535 (nothing is masked) and flags are exactly 0 or 1. The grader reports the specific reason a submission was gated.\n2. Differential test: execute each program on the grader's golden simulator (regenerated from the seed) and compare the agent's final state BIT-EXACT -- every register, every flag, every word of the graded memory window.\n3. Score = fraction of programs that match exactly; full credit (1.0) only if EVERY program matches.\n4. Pass/Fail cutoff: Pass = 1.0; any score below 1.0 is a Fail. The grader returns an explicit \"pass\" boolean; the continuous score is recorded for diagnostics only.\n\nDifficulty is calibrated and verified (see docs/DIFFICULTY_PROFILE.md, reproducible via selfcheck.py and evaluations/agent_baselines.py): a memorized/textbook ISA scores 0; a competent agent that slips on a single rule (flag-under-predication, branch base, sign-extension width, multi-register memory semantics, the published initial flag state, or an unimplemented instruction) scores ~0.45-0.98 but 0/6 full-pass; only a fully spec-faithful simulator reaches 1.0 on all six public seeds in both difficulty modes. Instruction coverage is asserted, not assumed: selfcheck.py fails if any opcode in the spec goes untested on any seed, so leaving a single instruction unimplemented scores 0.960 and still fails every seed.\n\nVariants: a few public seeds are shipped as worked examples; THIS submitted instance was generated from a held-out seed that is not one of them, and the live graded instance should use a further fresh seed (a different mutated ISA), regenerated by the grader, so nothing can be reused or memorized.","software_and_version":"Python 3.11 (any Python 3.9+) + NumPy >= 1.24 — NumPy is the only third-party dependency, and it is used solely for the seeded random-number generator that produces each instance's mutated ISA and program suite. The golden simulator, the reference simulator and the grader are pure standard-library Python, so grading needs no numerical or HDL toolchain at all. Open-source, headless, self-contained; requirements.txt included. Professional equivalent: SystemVerilog + iverilog/Verilator (RTL) co-simulated against a Sail/Spike-style golden ISA model, with riscv-dv-style conformance / random-instruction testing.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"fc962109-b38f-4045-b784-7a518f0b7bab","filename":"isa_spec.json","category":"input_materials","content_type":"application/json","size_bytes":8079},{"file_id":"a1e2ceb0-d675-4e7a-9e47-a3bc5d78e8d6","filename":"programs.json","category":"input_materials","content_type":"application/json","size_bytes":80918},{"file_id":"7c12e8f5-326e-4f4d-9f97-561da7968f2d","filename":"rx16_reference_and_evaluation(2).zip","category":"reference_output","content_type":"application/zip","size_bytes":128814}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally scoped, objectively verifiable, and reachable end to end with no material defects.\n\n**Completeness Check:**\n- Instruction: Complete — The ISA, encoding formats, instruction semantics, predication, flags, branch and multi-register toggles, quiescence rules, and exact output contract are fully specified.\n- Inputs: Complete — The ISA specification is self-contained, and all 25 programs provide valid instruction words, eight registers, 256 memory words, and initial Z/N/C/V flags; all 19 defined opcodes appear in the suite.\n- Toolchain: Complete — Linux, Python 3, and NumPy >=1.24 provide an open, headless, single-machine workflow with modest resource requirements.\n- Reference output: Complete — A structurally valid 25-entry reference result and a 25/25 passing grade report match the submitted input copies; independent reference and golden simulator implementations are included as evaluator evidence.\n- Rubric: Complete — Structural gates, exact integer ranges, bit-exact differential comparison, partial score calculation, and the 1.0 pass threshold are explicit and reflected in the static grader implementation.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. The inspected files were `isa_spec.json`, `programs.json`, and `rx16_reference_and_evaluation(2).zip`. Their byte counts and declared formats were consistent; both standalone JSON files parsed successfully, and the ZIP was readable, unencrypted, CRC-clean, free of traversal or link entries, and modest in expanded size. All archive members were inventoried, with relevant JSON, documentation, staging code, simulator implementations, grader logic, requirements, examples, and reports inspected. No manifest attachment was left unreviewed. Uploaded Python code was not executed under the review safety policy, so claimed self-test runs were assessed through static code and submitted reports rather than independently reproduced. Static staging logic exposes only the two input JSON files and keeps the exact submitted answers and evaluator implementation out of the solver environment. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n- `public_benchmark` — The package explicitly identifies public example variants and an Agents' Last Exam integration. Contamination is materially mitigated because live evaluation uses a fresh held-out seed that mutates the ISA itself, while reference outputs and generator state remain evaluator-only. External publication status was not independently verified beyond the submitted evidence.\n\n**Recommended Changes:**\nThe submission already provides unusually strong specification, reference, integration, and grading evidence. No material changes are required. As optional hardening, the evaluator could reject a bare top-level result list so enforcement exactly matches the documented `{\"results\": [...]}` envelope, and the harness adapter should receive the already-noted integration dry run before deployment.","reviewed_at":"2026-08-20T13:27:15.376190Z"},"claim":null}