{"submission_id":"cc61a74e-a6bd-4d5c-b325-847fff10f804","submission_revision_id":"f8a0fb3e-370d-40d1-9ec3-9ce2aaf033f4","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-13T19:32:57.918451Z","submitter_name":"Yanxin Zhang","submitter_email":"yzhang2879@wisc.edu","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"`claude-sonnet-4-6`  `gpt-5.5`","harness":"agent ,evaluate in  A100-PCIE-40GB","estimated_score":"40–60%"}},"submitted_at":"2026-08-13T19:32:57.918451+00:00","industry_domain":"Semiconductor & Microelectronics Design — GPU kernel engineering for quantised LLM inference","task_short_name":"w4a16_fused_gemm_triton_instance_1","operating_system":"Linux","task_description":"Objective: Reconstruct a 4-bit weight-only GEMM kernel from recordings of its results. The quantisation format is not supplied.\n\nStarting State: A quantised inference stack is in production. Its weight-only 4-bit GEMM kernel is the hot path at batch size one: every linear layer pulls its whole weight matrix across the bus to produce a single row of output, and four-bit weights cut that traffic to a quarter.\n\nThe kernel's source is gone, and so is the specification of the format it consumed. What survives is five recorded inference problems, each an .npz holding the fp16 activations x, the packed int32 qweight, the fp16 scales, the uint8 qzeros, the correct fp32 output y, and the group size. There is no description of how the packed tensors combine into a weight matrix.\n\nInstructions & Rules: Recover the packing layout and dequantisation rule, then write output/kernel.py exposing w4a16_matmul(x, qweight, scales, qzeros, group_size) as a Triton kernel that unpacks and scales inside the k-loop and never materialises the dequantised weight matrix.\n\nTwo of the five examples exist to make recovery possible rather than to be useful workloads: one has a square, well-conditioned x, so the weights come back by inversion; another is tall, so they come back by least squares as a cross-check. Those two use DIFFERENT group sizes, and so do the graded problems, so a rule that holds for one group size has not been recovered.\n\nDequantising to a full fp16 matrix and calling torch.matmul is numerically correct and far too slow to pass: it moves an order of magnitude more memory than it needs to.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"On the GPU in the VM, against six problems fixed in the grader and never shown — decode (M=1) and small batch, K and N from real 7B-to-13B transformer layers, group size varying between 64 and 128. Inputs come from a fixed seed.\n\nHARD GATES:\n- either deliverable missing, or kernel.py fails to import\n- kernel.py does not define a callable w4a16_matmul\n- the kernel raises, returns a non-tensor, or returns the wrong shape on any graded problem\n- relative Frobenius norm error against an fp32 reference above 5e-3 on any graded problem\n- median latency below 2.5x the unfused dequantise-then-matmul path on any graded problem\n\nSCORE: 0.60 x fraction of problems meeting the accuracy threshold, plus 0.40 x mean of min(1, speedup / 2.5). Passing requires every problem to clear both.\n\nBecause the correctness gate sits behind the recovery, getting the format wrong fails outright regardless of how good the kernel is.\n\nWHY THE FORMAT IS RECOVERABLE — verified, not assumed: solving the task from the shipped examples only, the square probe's x is well conditioned, so W = x^-1 y returns the dequantised weights with a residual of 2.5e-16; dividing by the given scales and adding the given zero points lands on integers 0..15 to within 0.004; the nibble at bit offset 4j is then confirmed to be the weight for k = 8*row + j. All five shipped examples are reproduced exactly (relative error 0.00e+00) by the grader's own interpretation rule, which matters because the example generator and the grader are separate code paths.\n\nTIMING: baseline and submission are timed in the same process on the same device, medians over 40 iterations after 10 warmups, so the threshold does not depend on how fast the grading machine is.\n\nWHY THE BAR IS ONLY 2.5x: the ratio does depend on GPU architecture, and this was measured. A reference kernel clears the baseline by 6.53x-11.87x on the L4 this task declares and by 3.12x-5.07x on an A100 unchanged, because the unfused path is bandwidth-bound. An earlier bar of 5x, set from the L4 numbers alone, made the reference kernel FAIL ITS OWN GRADER on the A100. A submission that simply calls the unfused path is numerically perfect and still fails, measuring 0.56x-0.91x.","software_and_version":"Python 3; PyTorch 2.x with CUDA; Triton 3.x; NumPy. All present in the gpu-free VM image; none redistributed with the task.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"5681d4e1-47f3-425e-886f-24abd0e847d9","filename":"B1_w4a16_fused_gemm_triton_INPUT.zip","category":"input_materials","content_type":"application/zip","size_bytes":478734},{"file_id":"6a0823d6-d8c5-45ef-92a4-e68c8c9b19b7","filename":"B1_w4a16_fused_gemm_triton_REFERENCE.zip","category":"reference_output","content_type":"application/zip","size_bytes":19899},{"file_id":"08e1cd8e-84af-4906-9ce4-493503df38aa","filename":"w4a16_fused_gemm_triton_selftest.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":3088}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, professionally scoped, objectively gradable, and reachable, with one deterministic evaluator-wiring correction needed during deployment.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly defines format recovery, the fused Triton implementation, required function signature, prohibited unfused approach, and two deliverables.\n- Inputs: Complete — Five valid NPZ cases contain all six promised arrays with consistent dtypes, dimensions, finite numeric data, and group sizes 64 and 128.\n- Toolchain: Complete — Linux, Python 3, CUDA-enabled PyTorch, and Triton form a legally obtainable single-machine toolchain; declared licenses and GPU requirements are adequate.\n- Reference output: Complete — Includes the quantization oracle, fused reference kernel, deterministic generator, current varied-group grader, and measurements from L4 and A100 hardware. A stale duplicate grader is present but does not require contributor-owned information to correct.\n- Rubric: Complete — Defines import and shape gates, relative-error threshold, per-case speed threshold, partial-score formula, hidden workload coverage, warmups, iterations, and median timing methodology.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. I inspected `B1_w4a16_fused_gemm_triton_INPUT.zip`, including its instructions, JSON manifest, and all five nested NPZ containers; `B1_w4a16_fused_gemm_triton_REFERENCE.zip`, including both graders, format implementation, reference kernel, generator, measurements, task card, and evaluator configuration; and `w4a16_fused_gemm_triton_selftest.txt`. Both ZIP archives and every nested NPZ passed structural and CRC checks, were unencrypted, and contained no unsafe paths, links, device entries, or excessive expansion. No manifest attachment was left uninspected. Python/NumPy and GPU execution were unavailable within the safe review path, so numerical reconstruction and latency measurements were not independently rerun; array headers, primitive payloads, finite-value status, representative values, and internal consistency were checked statically. The completion path is to recover the format from the two probes, implement `output/kernel.py`, document the work in `output/notes.md`, and evaluate six protected cases against the supplied fp32 oracle and colocated unfused baseline. The configured evaluator currently selects a stale `reference/grade.py` that looks for the hidden format module in solver inputs and fixes group size at 128; the supplied `scripts/grade.py` contains the correct protected-reference lookup and 64/128 cases, making this a routine, deterministic evaluator-wiring correction. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports an AI-synthetic, public-benchmark, or synthetic-defect-fixture label. Public identifiability was not externally searched because that is outside the permitted evidence boundary.\n\n**Recommended Changes:**\nThe recovery probes, hidden multi-group workloads, architecture-aware speed threshold, and self-test evidence are substantively sound. No contributor revision is required. During evaluator integration, stage the supplied current grader as the protected reference grader, or update the evaluator selector to use it, and remove the stale duplicate to prevent all submissions from failing before evaluation.","reviewed_at":"2026-08-14T20:41:18.144428Z"},"claim":null}