{"claim":{"id":"a3541c40-49dc-4ee8-9890-df0e2c86dc16","submission_id":"e544706a-4ce7-4fb8-ab25-98e34276812b","submission_revision_id":"2c30592f-08c7-4ff1-9b31-36afc27384d9","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:40:26.718708Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"e544706a-4ce7-4fb8-ab25-98e34276812b","submission_revision_id":"2c30592f-08c7-4ff1-9b31-36afc27384d9","revision_version":1,"submission_status":"under_review","submitted_at":"2026-07-07T20:27:12.291831Z","submitter_name":"Juyi Lin","submitter_email":"luke.juyi.lin@gmail.com","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"sonnet 5","harness":"claude code","estimated_score":">60%"}},"submitted_at":"2026-07-07T20:27:12.291831+00:00","industry_domain":"Machine Learning Infrastructure / LLM Inference Serving","task_short_name":"llm_serving_vllm_instance_1","operating_system":"Linux","task_description":"Objective: Root-cause and fix vLLM 0.20.2 so that Qwen3-Omni online serving with --mm-processor-kwargs '{\"use_audio_in_video\": true}' (the model card's documented launch command) actually works end-to-end: the server starts, the audio track of uploaded videos genuinely reaches the model, and videos without an audio track degrade gracefully instead of crashing the engine.\n\nStarting State: A Linux box (Ubuntu 22.04) with at least one ~94 GB GPU (minimal: 1× H100 NVL, -tp 1; reference run used 2× H100 NVL with -tp 2), a Python venv with vllm==0.20.2, transformers==5.13.0, torch==2.11.0+cu130 installed from pip, and the Qwen/Qwen3-Omni-30B-A3B-Instruct weights pre-downloaded (65.68 GiB checkpoint; weights need ~62.4 GiB VRAM total). Input files: test_withaudio.mp4 (125.6 s football-match clip with English commentary audio track), test_noaudio.mp4 (the same clip with the audio track stripped via ffmpeg -c:v copy -an; video stream byte-identical), and probe_audio_ab.py (an A/B client that posts both videos as base64 video_url chat completions and records prompt_tokens + the answer for each).\n\nInstructions & Rules:\n\nLaunch vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --mm-processor-kwargs '{\"use_audio_in_video\": true}' -tp <number of GPUs> and observe that it is broken (as of 0.20.2 the engine crashes during startup profiling with a StopIteration inside the HF processor).\nDiagnose and fix vLLM (editing the installed package tree is fine) until ALL of the following hold with a server launched by the command above:\nthe server starts and /health returns 200;\nprobe_audio_ab.py shows test_withaudio.mp4 consuming strictly more prompt_tokens than test_noaudio.mp4, and the with-audio answer quotes the spoken commentary while the no-audio answer is NO_SPEECH (i.e. audio genuinely reaches the model via the server-level flag — standard OpenAI clients do not send per-request mm_processor_kwargs);\na request with per-request {\"mm_processor_kwargs\": {\"use_audio_in_video\": true}} and the with-audio video returns HTTP 200 (no placeholder-count RuntimeError);\nsending test_noaudio.mp4 (no audio track at all) returns HTTP 200 and the engine stays alive afterwards (/health still 200) — including when it is sent AFTER test_withaudio.mp4 in the same server lifetime (same pixels, different audio pairing must not poison any cache).\nDeliverables (exact paths): output/submission/fix.patch — a unified diff of every vLLM file you changed, applying cleanly (git apply --check) to a pristine pip-installed vllm 0.20.2 tree; output/submission/probe_ab_result.json — the JSON written by running probe_audio_ab.py against your fixed server; output/submission/server.log — the stdout/stderr of the fixed server covering startup through all probe requests.\nDo not modify probe_audio_ab.py, the input videos, or the model weights. Do not fork to a different vLLM version; the fix must be a patch on 0.20.2.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"All checks are automated; the grader has a pristine copy of the pip-installed vllm 0.20.2 tree and the two input videos.\n\nPatch exists and applies: output/submission/fix.patch exists, is a valid unified diff, and git apply --check fix.patch succeeds against the pristine vllm 0.20.2 package tree (exit code 0). Only files under vllm/ may be touched; the patch must not modify probe_audio_ab.py or any test asset.\nStartup crash fixed: apply the patch to the pristine tree, launch vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --mm-processor-kwargs '{\"use_audio_in_video\": true}' -tp <number of GPUs> (-tp 1 on the minimal single-GPU VM); GET /health must return HTTP 200 within 20 minutes of launch (reference startup on 2× H100 NVL is well under this).\nAudio actually fed (server-level flag): run the unmodified probe_audio_ab.py against the patched server; in the resulting JSON: with_audio.prompt_tokens - no_audio.prompt_tokens >= 500 (reference: 13194 − 11559 = 1635), no_audio.answer contains the substring NO_SPEECH, and with_audio.answer does NOT contain NO_SPEECH and contains, case-insensitively, at least 2 of the commentary-only keywords {\"penalty\", \"handball\", \"offside\", \"goalkeeper\", \"VAR\", \"referee\"}.\nPer-request kwargs path fixed: POST the with-audio video with \"mm_processor_kwargs\": {\"use_audio_in_video\": true} in the request body → HTTP 200 (no Expected there to be N prompt placeholders RuntimeError).\nGraceful degradation + engine survival: in the SAME server lifetime and in this order — send test_withaudio.mp4, then test_noaudio.mp4 (both plain requests, server-level flag only). Both must return HTTP 200, and GET /health must return 200 after the second response (guards the engine-killing cached-item/mRoPE NoneType crash and the sync fetch_audio HTTP 500 on audio-less videos).\nSubmitted artifacts match a live run: output/submission/probe_ab_result.json exists, is valid JSON with keys with_audio, no_audio, prompt_tokens_differ: true, and its prompt_tokens values satisfy check 3; output/submission/server.log exists, is non-empty, and contains no Python traceback after the \"server ready\" line.","software_and_version":"vLLM 0.20.2 (pip install), transformers 5.13.0, torch 2.11.0+cu130 ","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"2d927de6-ad11-4d88-a584-5e9d35b31ddf","filename":"reference_output.zip","category":"reference_output","content_type":"application/zip","size_bytes":14569},{"file_id":"12ae243f-fb21-4164-931e-9e3ca9b6220c","filename":"input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":27006016}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The task is complete, technically rigorous, objectively gradable, and feasible on the declared GPU environment with coherent input and reference evidence.\n\n**Completeness Check:**\n- Instruction: Complete — The defect, required behavior, constraints, exact deliverables, and success conditions are explicit.\n- Inputs: Complete — Both 125.6-second, 1280×720 AVC videos are valid; their encoded video payloads are identical, while only the with-audio variant contains an AAC track. The probe source matches the described A/B workflow.\n- Toolchain: Complete — Exact package versions, operating system, GPU capacity, model, launch command, and permitted patch target are specified.\n- Reference output: Complete — The reference contains a valid three-file vLLM-only unified patch, schema-valid probe results, and a server log showing successful startup and four HTTP 200 inference requests without a post-startup traceback. The result has a 1,635-token delta, `NO_SPEECH` for silent video, and all six specified commentary keywords for audio video.\n- Rubric: Complete — Automated checks independently cover patch applicability, startup, audio delivery, per-request kwargs, cache isolation, graceful degradation, engine survival, and artifact integrity.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both `reference_output.zip` and `input_materials.zip` were fully inspected and were readable, unencrypted, CRC-clean ZIP containers with safe relative entries and modest expansion. Substantive members inspected were `fix.patch`, `probe_ab_result.json`, `server.log`, `probe_audio_ab.py`, `test_withaudio.mp4`, and `test_noaudio.mp4`. Only non-substantive AppleDouble metadata sidecars were not content-decoded. The reference archive is separately designated as reference evidence and should remain evaluator-only. The review environment lacked the pristine vLLM package tree and required GPU/model runtime, so clean application and live serving were not independently rerun; patch syntax and scope were validated statically. Completion and evaluation remain direct: patch the pristine 0.20.2 tree, launch the declared server, query health, execute the A/B and per-request tests in the required order, verify continued health, and validate the three output artifacts. The probe’s model-suffixed result filename requires only deterministic output-location normalization. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission provides representative inputs, a credible reference implementation, live-run evidence, and unusually precise regression criteria. No changes are required.","reviewed_at":"2026-08-15T00:01:55.522491Z"},"claim":{"id":"a3541c40-49dc-4ee8-9890-df0e2c86dc16","submission_id":"e544706a-4ce7-4fb8-ab25-98e34276812b","submission_revision_id":"2c30592f-08c7-4ff1-9b31-36afc27384d9","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:40:26.718708Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}