{"claim":{"id":"d76e57ec-7961-4e06-85ee-9c40ca796c79","submission_id":"26e07878-f150-4e63-bd92-6c9346e557f0","submission_revision_id":"727d6c38-5ad0-4718-9e49-dd6723cef86d","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:18.387281Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"26e07878-f150-4e63-bd92-6c9346e557f0","submission_revision_id":"727d6c38-5ad0-4718-9e49-dd6723cef86d","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-25T02:29:21.762111Z","submitter_name":"JICHENG WANG","submitter_email":"jconliner67@gmail.com","task":{"status":"submitted","version":3,"extra_fields":{},"submitted_at":"2026-08-25T02:29:21.762111+00:00","industry_domain":"Sports & Broadcast Analytics / Computer Vision Engineering","task_short_name":"dancetrack_broadcast_mot_1","operating_system":"Linux","task_description":"Objective: Acting as a computer-vision engineer at a sports and entertainment analytics company, build a complete multi-object tracking pipeline and deliver per-dancer tracks for group dance footage: for every video frame, a bounding box around each dancer with a track ID that stays consistent for the same person across the whole clip. Downstream products (choreography analytics, per-performer highlight reels, motion statistics) consume these tracks, so identity switches are the costly failure mode.\n\nStarting State: You are given the raw frames (JPEG, about 1280x720, 20 fps) and seqinfo.ini for exactly 5 held-out sequences of the DanceTrack validation split (CVPR 2022 benchmark, https://dancetrack.github.io) - dancetrack0004 (1203 frames), dancetrack0025 (803), dancetrack0041 (1003), dancetrack0047 (1203), dancetrack0073 (703); 4,915 frames, about 470 MB - laid out MOT-style as data/val/<seq>/img1/%08d.jpg plus data/val/<seq>/seqinfo.ini, with NO ground truth included. No detector, no tracker and no environment are set up.\n\nDanceTrack is deliberately hard for this workflow: dancers wear near-identical outfits, so appearance features barely help, and they move non-linearly with frequent position swaps and occlusions, so naive frame-to-frame matching produces heavy identity churn. Producing a professional-quality deliverable requires real engineering: choosing and obtaining a person detector, building or adapting a motion-model-based association stage (Kalman-filter-style SORT/ByteTrack/OC-SORT lineage or better), handling occlusion re-entry, and validating the output format end to end. Several hours for an experienced MOT engineer; days for a generalist.\n\nInstructions & Rules:\n1. Obtain the 5 sequences' frames with the provided stdlib-only script download_inputs.py, which fetches ONLY these sequences' image frames and seqinfo.ini from the official Hugging Face mirror (noahcao/dancetrack) via HTTP range requests (~470 MB transfer, no account needed). It selects members statically and never touches the gt/ directories. This is the only sanctioned retrieval path: do NOT download the full validation archive, because it carries the validation ground truth that the integrity rule forbids you to obtain. If the mirror is unavailable, stop and report that rather than substituting another source - the evaluator can stage the frames instead.\n2. INTEGRITY RULE (auto-fail if violated): the DanceTrack val annotations are public. You must NOT download, read, or use ground-truth annotations for any DanceTrack val sequence anywhere in your pipeline, and you must not use any model trained or tuned on the DanceTrack val or test splits. Models trained on the DanceTrack TRAIN split (e.g. the officially released ByteTrack/YOLOX weights) and any COCO/CrowdHuman/MOT17-trained detectors are allowed. Compliance is audited from your working transcript.\n3. Software choice is otherwise free: any open-source language, framework, or pretrained weights; internet access to download code and weights is allowed.\n4. Deliver a submission directory with exactly one plain-text file per sequence, named after the sequence (dancetrack0004.txt, dancetrack0025.txt, dancetrack0041.txt, dancetrack0047.txt, dancetrack0073.txt). Each line is MOTChallenge format:\n   <frame>,<id>,<bb_left>,<bb_top>,<bb_width>,<bb_height>,<conf>,-1,-1,-1\n   Every line must have EXACTLY 10 comma-separated fields, all finite numbers. `frame` is 1-indexed and must match img1/%08d.jpg and the sequence's seqLength; `id` is a STRICTLY POSITIVE integer track identifier (id > 0) that stays consistent across frames for the same dancer; box coordinates are pixels in the original image resolution with positive width and height; `conf` is the detection confidence (use 1 if not applicable). One line per tracked person per frame. These are hard format gates: a violation fails the submission before any metric is computed.\n5. Compute budget: fits within about 1 GPU-hour (detector inference over 4,915 frames; association is CPU-trivial), or a few CPU-hours with a lighter detector.\n\nEvaluator / solver separation: ground truth, oracle tracks and the grading script are evaluator-only and are never part of the solver's inputs. The recommended execution configuration is to stage the frames, let the solver fetch its own code and pretrained weights, then run scoring in a separate step that has the ground truth and the solver's output but where the solver no longer runs. Because DanceTrack validation annotations are publicly retrievable, running the solving step offline (after code and weights are staged) gives the strongest integrity; the transcript audit is the fallback when network access is left open.\n\nLicense note for reviewers: DanceTrack annotations and the Hugging Face mirror are distributed under CC-BY-4.0 (dataset cards of noahcao/dancetrack and Voxel51/DanceTrack); the video frames are released for non-commercial research use only (DanceTrack project page and paper, Sun et al., CVPR 2022). The reference bundle redistributes only the 5 sequences' ground-truth text files (CC-BY-4.0, with attribution); image frames are downloaded directly from the official mirror and are not redistributed. Usage within ALE is research-only evaluation, consistent with both licenses.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Fully automated metric comparison against held-out ground truth, using the official MOT evaluation toolkit. No human judgement, no LLM judging. Usage: python grade.py <submission_dir>; CPU-only, under a minute, needs Python 3.9+, numpy and scipy.\n\nStep 1 - hard format gates. Any of these fails the submission with SCORE 0 before metrics are computed: a missing per-sequence file, a line that does not have exactly 10 comma-separated fields, a non-finite value, a frame index outside [1, seqLength] for that sequence, a track id that is not strictly positive (id > 0), or a non-positive box width/height. These gates are enforced exactly as the task statement words them; the packaged oracle reference tracks satisfy them too (their IDs are 1-based).\n\nStep 2 - scoring. The grader runs the vendored official TrackEval toolkit, the identical protocol and metric implementation used by the DanceTrack benchmark, against the withheld validation ground truth for the 5 sequences (shipped in the evaluator-only bundle under grading/gt/, never given to the solver), and reports HOTA (primary), DetA, AssA and IDF1 combined over all sequences.\n\nFinal score: score = clip((HOTA - 30.0) / (50.0 - 30.0), 0, 1). HOTA at or below 30 scores 0, at or above 50 scores 1, linear in between.\n\nThreshold justification from published anchors and our own harness calibration:\n- ByteTrack on the full DanceTrack val split reaches HOTA 47.1 (AssA 31.5, IDF1 51.9) per the official DanceTrack repository, so a strong expert pipeline scores about 0.855.\n- OC-SORT reaches HOTA 55.7 on the DanceTrack test split (official leaderboard) and about 52 on val, so OC-SORT-class association engineering earns full credit at the target of 50.\n- Published DanceTrack baselines cluster at HOTA 45-55 (DeepSORT 45.6, ByteTrack 47.3, SORT 47.9, MOTR 54.2).\n- Calibration we ran on these exact 5 sequences with the attached harness: perfect tracks give HOTA 100.00 (score 1.0); perfect boxes with per-frame scrambled IDs give HOTA 3.51 (score 0); even perfect ground-truth boxes with greedy IoU association only reach AssA 52.4 / HOTA 72.31 - and no solver has ground-truth boxes.\nThe floor of 30 therefore sits below anything a working expert pipeline produces and above degenerate output, so partial credit tracks real quality and full credit requires genuine state-of-the-practice association engineering rather than just a good detector.\n\nNote on fabrication resistance: unlike a task whose deliverable is a handful of numbers, the required artifact here is roughly 46,000 rows of per-frame tracking output scored against ground truth the solver does not have. Knowing the published baselines does not let anyone produce it, so the deliverable is itself the evidence that the work was done.\n\nIntegrity check: because DanceTrack val ground truth is publicly downloadable, the score is only valid if the solver did not use it. Evaluation is invalid (score 0) if the working transcript shows val or test ground truth was accessed, or that a model trained or tuned on the DanceTrack val/test splits was used. Models trained on the DanceTrack train split or on COCO/CrowdHuman/MOT17 are allowed. Running the solving step without network access, after code and weights are staged, removes the need to rely on the audit.","software_and_version":"Python 3.9+, PyTorch 2.x, open-source detector+tracker stack (e.g. YOLOX / ByteTrack / OC-SORT), TrackEval, numpy, scipy","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"7bc67244-771a-4a40-a4be-878981cd5eb3","filename":"03_input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":5055},{"file_id":"8a258b30-b996-48b3-82d8-e9dcc04864af","filename":"03_reference_output_and_grader.zip","category":"reference_output","content_type":"application/zip","size_bytes":1527842}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, professionally scoped, objectively gradable, and reachable, with public-benchmark leakage appropriately addressed through evaluator isolation and offline solving.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, integrity constraints, five sequences, output schema, compute budget, and completion criteria are explicit and internally consistent.\n- Inputs: Complete — The solver bundle defines the exact 4,915-frame input set and includes a standard-library downloader that statically excludes ground-truth directories; remote availability was not independently tested under the review evidence boundary.\n- Toolchain: Complete — A legally obtainable Python, detector, tracker, and CPU/GPU execution path is described without requiring a particular proprietary implementation.\n- Reference output: Complete — Evaluator-only ground truth, five oracle outputs, five naive-association calibration outputs, sequence metadata, attribution, and vendored evaluation code are present. All 46,283 rows in each reference-output set have consistent numeric schemas; oracle IDs are correctly shifted from zero-based ground-truth IDs to positive submission IDs.\n- Rubric: Complete — The grader implements the stated format gates, official combined-sequence HOTA/DetA/AssA/IDF1 evaluation, and the documented clipped linear score.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`.\n\nBoth `03_input_materials.zip` and `03_reference_output_and_grader.zip` were inspected. Their downloaded sizes matched the manifest. Both were readable, unencrypted ZIP archives with valid container integrity, no unsafe paths, links, device entries, or excessive expansion. The input archive contained two UTF-8 text members: the solver statement and downloader source. The reference archive contained 136 members, including the grader, all five ground-truth and sequence-metadata sets, ten reference tracker files, attribution and license material, and vendored TrackEval source. All members were inventoried and typed; grading-relevant source and representative schemas were inspected. Ancillary TrackEval documentation, tests, and unrelated dataset adapters were inventoried but not reviewed line by line. Uploaded programs were not executed, and external mirrors, project pages, licensing claims, and published benchmark scores were not independently accessed.\n\nThe end-to-end path is coherent: obtain or stage only the five image sequences, run an allowed person detector and association tracker, emit five MOTChallenge files, apply deterministic format validation, then evaluate against isolated ground truth using combined HOTA and identity metrics. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — DanceTrack and its validation annotations are explicitly identified as public, making protected answers network-retrievable. The submission materially mitigates this through separate evaluator-only packaging, a downloader that excludes ground truth, transcript auditing, and a recommended workflow that stages allowed inputs and weights before disabling solver network access. External provenance and benchmark claims could not be independently checked within the evidence boundary.\n\n**Recommended Changes:**\nThe submission is ready and requires no material contributor changes. Operationally, keep the reference archive entirely evaluator-only and disable solver networking after staging frames and allowed code and weights. Optional grader hardening would reject extra submission files, enforce `-1` in the final three fields, cap file sizes and track-ID magnitude, and verify exact HTTP range responses in the downloader.","reviewed_at":"2026-08-25T02:39:51.045941Z"},"claim":{"id":"d76e57ec-7961-4e06-85ee-9c40ca796c79","submission_id":"26e07878-f150-4e63-bd92-6c9346e557f0","submission_revision_id":"727d6c38-5ad0-4718-9e49-dd6723cef86d","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:11:18.387281Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}