{"claim":{"id":"3fb7d317-493b-416b-92da-040e562567b1","submission_id":"7f7261b5-01a4-483a-9e7b-cbbcc790e65a","submission_revision_id":"ddb6d1d7-a6c5-4e75-b915-6375f93f9a4b","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:59.224169Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"7f7261b5-01a4-483a-9e7b-cbbcc790e65a","submission_revision_id":"ddb6d1d7-a6c5-4e75-b915-6375f93f9a4b","revision_version":10,"submission_status":"under_review","submitted_at":"2026-07-25T01:43:12.036120Z","submitter_name":"Yunzhe Wang","submitter_email":"yunzhewa@usc.edu","task":{"status":"submitted","version":10,"extra_fields":{"difficulty_self_test":{"model":"gpt-5.6-sol-xhigh","harness":"codex","estimated_score":"<1%"}},"submitted_at":"2026-07-25T01:43:12.036120+00:00","industry_domain":"Game Development / 3D Voxel Modeling","task_short_name":"minecraft_video_build_reconstruction_1","operating_system":"Windows, Linux, MacOS","task_description":"# Minecraft Building Reconstruction\n\n## Task\n\nWatch the complete `tutorial.mp4` video and recreate the final Minecraft building exactly.\nThe video is self-contained, no audio, listed all the materials needed, and shows the exact building process, with a front, left, right, back, and top overview at the end.\nThe building is a modern house of size 10x10x10. The ground layer is excluded.\n\nMatch every block's:\n\n- Material\n- Relative position\n- Block state, including stair direction, slab type, door hinge, log axis, and other orientation properties\n\nSave the completed building as a valid Sponge v3 schematic of size 10x10x10 at:\n\n```text\noutput.schem\n```\n\nOnly `output.schem` will be submitted. \n\n## Scoring\n\n`output.schem` is compared with a hidden ground-truth schematic.\n\n- A score of `1.0` means every non-air block and block state matches exactly.\n- Missing, extra, incorrect, or incorrectly oriented blocks reduce the score.\n- Translation and air padding are ignored.\n- Rotation and mirroring are not allowed.\n- Schematic metadata, version, entities, and block-entity data are not scored.\n\n## Rules\n\n- Use `tutorial.mp4` as the only reference for the building.\n- Do not use web search.\n- Do not access or modify the hidden ground truth or scoring files.\n- Do not include terrain, scaffolding, markers, or temporary blocks unless they are part of the final build.\n- Verify that `output.schem` is valid and readable before finishing.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Since the output is deterministic, we can evaluate it though a script and the agent must produce a output building that matches the reference exactly.\n\nI made an evaluator script for this exact purpose and you can run it with:\n\npython3 score_schematics.py output.schem answer.schem --compact\n\nThe final score is a partial-credit Intersection over Union (IoU) score over exact non-air block states. Specifically, \n\nLet:\nG = the set of ground-truth elements\nC = the set of candidate elements\n\nEach element is:\n(relative_x, relative_y, relative_z, canonical_block_state)\n\nThe score is:\nscore = |C ∩ G| / |C ∪ G|\n\nThe result ranges from 0 to 1:\n\n- 1.0 means every non-air block, coordinate, material, and block-state property matches exactly.\n- A value between 0 and 1 awards partial credit according to the proportion of exact matches.\n- 0 means there are no exact matching block-state elements.\n\nMissing blocks and extra blocks increase the union without increasing the intersection. A block at the correct position but with the wrong material or state—such as a stair facing the wrong direction—is treated as a mismatch and receives no exact-match credit.\n\nThe evaluator ignores air blocks, air padding, schematic dimensions, offsets, author metadata, timestamps, data-version metadata, entities, and block-entity payloads. Translation is normalized, but rotation and mirroring are not allowed.\n\nThe JSON field named \"score\" is the final task score. Missing, unreadable, or invalid output.schem files receive a score of 0.\n\nSince the building is small, and the video shows the exact process, a trained human should be able to easily archive a score of 1. \n\nConsidering some flexibility in interior design and decorating block placement, we can tolerate variations and define > 0.90 as a passing score, which corresponds to about 25 blocks that is of wrong type/state or misplaced.\n","software_and_version":"Python 3.12, FFmpeg 7, nbtlib 1.0.2, Sponge Schematic v3","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"b781a42b-4def-459f-8ea9-df169ffa4f89","filename":"tutorial.mp4","category":"input_materials","content_type":"video/mp4","size_bytes":157015574},{"file_id":"a5416caf-124a-4a76-87c6-5d1de27244e4","filename":"score_schematics.py","category":"reference_output","content_type":"application/octet-stream","size_bytes":14993},{"file_id":"8f789b5d-8c0b-48fe-85c8-51f735793e31","filename":"codex-gpt5.6sol-xhigh.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":297968314},{"file_id":"444d924d-5fc2-4c13-9732-871b120370d2","filename":"codex-gpt5.6sol-high.zip","category":"difficulty_self_test_evidence","content_type":"application/zip","size_bytes":286120430},{"file_id":"1110b076-749d-46df-b69a-09ec6bec7c7c","filename":"readme.md","category":"difficulty_self_test_evidence","content_type":"text/markdown","size_bytes":956},{"file_id":"dfe62a85-eda5-4c1d-872c-31f87f7a7425","filename":"answer.schem","category":"reference_output","content_type":"application/octet-stream","size_bytes":817},{"file_id":"ccfb0fcd-017f-49cb-a525-69d472f54ada","filename":"instruction.md","category":"input_materials","content_type":"text/markdown","size_bytes":1415}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, objectively scorable, and end-to-end reachable, with only nonmaterial documentation inconsistencies.\n\n**Completeness Check:**\n- Instruction: Complete — clearly specifies the 10×10×10 reconstruction, exact block-state requirements, orientation constraints, output format, and exclusions.\n- Inputs: Complete — the 393.459-second, 1240×720 H.264 tutorial is readable and representative frames confirm the material list, orientation, construction process, interior details, and final exterior views. It contains an AAC track despite being described as having no audio, but this does not affect the visual task.\n- Toolchain: Complete — Python, FFmpeg, and nbtlib are legally obtainable, cross-platform tools supporting the complete workflow.\n- Reference output: Complete — `answer.schem` is a valid gzip-compressed Sponge v3-shaped 10×10×10 NBT schematic containing 1,000 positions, 23 palette states, and 482 non-air blocks. The evaluator is coherent with the output contract.\n- Rubric: Complete — exact block-state IoU, translation handling, ignored data, failure behavior, and the passing threshold are explicitly defined.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`. I inspected `tutorial.mp4`, `instruction.md`, `score_schematics.py`, `answer.schem`, `readme.md`, `codex-gpt5.6sol-xhigh.zip`, and `codex-gpt5.6sol-high.zip`. Both ZIP directories were inventoried and showed no encryption, unsafe paths, symbolic links, device entries, or excessive expansion; score reports, trajectory summaries, valid JSONL trajectories, candidate schematics, reconstruction scripts, and five contact sheets were inspected without executing submitted code. Large duplicate MP4/MOV members, remaining frame images, and macOS metadata were inventoried but not decoded because they are redundant and noncritical. The review environment lacked Python and FFmpeg, so the submitted evaluator was not executed and media was not directly decoded; independent structural parsing reproduced both reported IoU scores and validated the schematics. The xhigh score report reverses candidate and ground-truth labels, but its symmetric IoU result remains correct. Solver inputs and reviewer-only reference/self-test materials are separated by manifest role; the self-test archives contain ground-truth copies and must remain reviewer-only. An ordinary user can inspect the tutorial, generate `output.schem` with the declared open-source toolchain, and receive deterministic evaluation against the withheld reference. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe core task, evidence, and evaluation path are valid. No material changes are required.","reviewed_at":"2026-08-14T22:08:10.414644Z"},"claim":{"id":"3fb7d317-493b-416b-92da-040e562567b1","submission_id":"7f7261b5-01a4-483a-9e7b-cbbcc790e65a","submission_revision_id":"ddb6d1d7-a6c5-4e75-b915-6375f93f9a4b","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:59.224169Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}