{"claim":{"id":"f18c172c-cf4b-45f0-a969-8884272101cd","submission_id":"938ae532-2eef-4d89-a3db-7e2366f813b0","submission_revision_id":"e78d383e-1d4c-47d3-bd23-9f507c3c65aa","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:25:07.815240Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"938ae532-2eef-4d89-a3db-7e2366f813b0","submission_revision_id":"e78d383e-1d4c-47d3-bd23-9f507c3c65aa","revision_version":1,"submission_status":"under_review","submitted_at":"2026-09-09T03:09:10.848284Z","submitter_name":"Yunze XIao","submitter_email":"yunzex@andrew.cmu.edu","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT 5.6 sol","harness":"Codex","estimated_score":"10–20%"}},"submitted_at":"2026-09-09T03:09:10.848284+00:00","industry_domain":"Mechanical Engineering / Manufacturing process planning","task_short_name":"mfcad_feature_labelling_instance","operating_system":"Linux","task_description":"Objective: Label every face of 30 B-rep parts with the machining feature it belongs to (through hole, rectangular pocket, chamfer, ... 24 feature classes) or mark it as untouched stock, the way a process planner does before deciding operations.\nStarting State: `input/parts/<part_id>.step` (30 STEP files; every face entity is named in file order f000, f001, ...), `input/examples/` (5 parts of the same kind with their full labels as `.labels.json`), `input/feature_classes.txt` (class id to name table, 0-23 features, 24 = stock), `input/README.md`.\nInstructions & Rules: Load each part with an open-source B-rep tool, inspect face geometry and topology, decide the class of every face, and write `output/labels.json` as {\"<part_id>\": {\"f000\": <int class>, \"f001\": <int class>, ...}, ...} covering all 966 faces. Integer classes only; use the face names from the STEP file as ids. No external datasets or pre-trained feature-recognition models may be downloaded; the five examples are the only labelled data.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Deterministic. Parse `output/labels.json`; if missing or not a JSON object the score is 0. For every (part, face) pair in `reference/labels.json` (966 pairs), compare the predicted integer class with the true class; a missing face counts as wrong. Compute macro-averaged F1 over the 25 classes present in the reference; score = macro-F1 clipped to [0, 1]. Calibration on the reference: perfect labels 1.0; labelling every face as stock 0.014; 80 % of faces correct 0.75. Implementation: `grade.py::score_labels`.","software_and_version":"Python 3.12 with CadQuery 2.x / OCP 7.7 (OpenCASCADE); FreeCAD 0.21 optional. Any B-rep tool that reads STEP AP214 works.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"eb2c485d-5d19-4983-8af6-d2824d260eaa","filename":"input.zip","category":"input_materials","content_type":"application/zip","size_bytes":753945},{"file_id":"3deff0e9-00af-4073-acbe-8f60694a5bdb","filename":"reference.zip","category":"reference_output","content_type":"application/zip","size_bytes":2840},{"file_id":"c7a64b15-0b16-4a79-adcb-e2698b2e40d4","filename":"trajectory.jsonl","category":"difficulty_self_test_evidence","content_type":"application/octet-stream","size_bytes":36467}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — complete, internally consistent, objectively evaluable, and reachable on a single open-source toolchain.\n\n**Completeness Check:**\n- Instruction: Complete — clearly defines the 25 classes, face-ID convention, 30-part scope, required JSON output, and labeling objective.\n- Inputs: Complete — `input.zip` contains the README, class table, five labeled examples, and all 30 ASCII AP214 STEP parts; face counts and example schemas are consistent.\n- Toolchain: Complete — Python, CadQuery/OCP, and compatible open-source B-rep alternatives are specified without credential or licensing barriers.\n- Reference output: Complete — `reference.zip` contains labels and a manifest covering all 30 parts and 966 faces, with valid integer classes 0–24.\n- Rubric: Complete — deterministic parsing, missing-face handling, macro-F1 scoring, class coverage, and calibration examples are specified.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. I inspected all members of `input.zip` and `reference.zip`, including the README, class table, all labeled examples, all 30 STEP files, reference labels, and reference manifest, plus all 26 records of `trajectory.jsonl`. Archives are parseable, unencrypted, free of unsafe member paths or special-file entries, and pass integrity tests. All STEP files are readable ASCII; every expected face sequence is present, and reference labels match the manifest counts. Uploaded trajectory commands were inspected as text and not executed; its final claims were not independently scored. No required content was left unreadable, and no material reviewer-tooling limitation affects the declared workflow. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. There is no affirmative evidence supporting AI-synthetic, public-benchmark, or synthetic-defect-fixture screening; public provenance was not independently verified.\n\n**Recommended Changes:**\nThe submission provides a valid, coherent input package, reference contract, toolchain path, and deterministic rubric. No changes are required.","reviewed_at":"2026-09-09T03:20:04.793012Z"},"claim":{"id":"f18c172c-cf4b-45f0-a969-8884272101cd","submission_id":"938ae532-2eef-4d89-a3db-7e2366f813b0","submission_revision_id":"e78d383e-1d4c-47d3-bd23-9f507c3c65aa","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:25:07.815240Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}