{"claim":{"id":"29685ccf-5a2e-407d-a8b2-8f555c7940c4","submission_id":"6fa6823d-49e4-46cd-bac1-fc3bc6b13434","submission_revision_id":"24dcc140-ea77-4d44-a2a2-eca694b250b0","builder":"builder-01","status":"active","claimed_at":"2026-10-15T16:20:51.918586Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"6fa6823d-49e4-46cd-bac1-fc3bc6b13434","submission_revision_id":"24dcc140-ea77-4d44-a2a2-eca694b250b0","revision_version":8,"submission_status":"under_review","submitted_at":"2026-10-11T03:28:08.987419Z","submitter_name":"Ipek Oztas","submitter_email":"ipek_oztas@brown.edu","task":{"status":"submitted","version":8,"extra_fields":{},"submitted_at":"2026-10-11T03:28:08.987419+00:00","industry_domain":"Computer Science (Visual Computing)","task_short_name":"real2sim_cloth_transfer_001","operating_system":"Linux, MacOS","task_description":"You are a robotics simulation engineer building a digital twin of a robot manipulating a cotton towel over a table. Use the supplied real 3D observations and nominal gripper commands to calibrate a MuJoCo cloth simulation. Match slow table contact and the observed beginning of a dynamic fling, then predict the remainder of that fling with the same physical parameters. Deliver the calibrated parameter file, editable model, complete simulated trajectories, execution log, and a short explanation of the calibration. The evaluator reruns the supplied physics model using the submitted parameters and compares it with withheld real measurements. Per-frame animation, state resets, and observation-driven corrections are not allowed.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Evaluate `answer.json` by rerunning the supplied canonical MuJoCo simulator. Compare the resulting cloth motion with the 70 withheld real point-cloud frames, IDs 60–129. Do not score submitted animations or compare parameter values with a presumed “true” parameter vector.\n\nFor each frame, calculate the Euclidean distance from every reference point to its nearest sampled simulated surface point. Use the supplied evaluator’s deterministic 16,384-point surface sampling. Compute the frame’s mean distance and 95th-percentile distance, then average each statistic equally across all 70 frames.\n\nAssign credit as follows:\n\n- **Mean-distance credit:** 1.0 at ≤20 mm; 0.0 at ≥80 mm; linear interpolation between them.\n- **95th-percentile credit:** 1.0 at ≤50 mm; 0.0 at ≥180 mm; linear interpolation between them.\n- **Final score:** 60% mean-distance credit + 40% 95th-percentile credit.\n\nFull success requires a valid simulation, mean error ≤20 mm, and average per-frame 95th-percentile error ≤50 mm.\n\nMissing required parameter keys, extra keys, nonnumeric or nonfinite values, out-of-range parameters, or unstable simulations receive zero. Evaluator infrastructure failures must be reported separately.\n\nUse the fixed geometry, controls, topology, solver settings, and parameter bounds supplied with the task. No additional alignment, time warping, state resets, or frame-dependent corrections are permitted. Withheld observations must remain inaccessible to the agent, including through retrieval from the public source dataset.","software_and_version":"MuJoCo 3.15.0, Python, NumPy, and SciPy. ","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"ede93ecf-d37f-46db-b728-c83599f96825","filename":"agent_input.zip","category":"input_materials","content_type":"application/zip","size_bytes":25295861},{"file_id":"c1795216-6c96-4d67-9639-2428b9ca3fc3","filename":"Dockerfile","category":"reference_output","content_type":"application/octet-stream","size_bytes":251},{"file_id":"6fca2415-b041-4d46-9ae4-448dea68a5fa","filename":"PREREQUISITES.md","category":"reference_output","content_type":"text/markdown","size_bytes":4144},{"file_id":"68ce0949-632c-4ac0-9b41-a7123de86cf9","filename":"verify_container.py","category":"reference_output","content_type":"text/x-python-script","size_bytes":2390},{"file_id":"7bc255e6-f209-4cf3-807d-9f5ce60e3f41","filename":"score.py","category":"reference_output","content_type":"text/x-python-script","size_bytes":1797},{"file_id":"a2213974-7418-47bd-8a09-6bfb9cde0b9a","filename":"ground_truth.npz","category":"reference_output","content_type":"application/octet-stream","size_bytes":3087871}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task provides coherent calibration inputs, a fixed simulation contract, and objective scoring, with explicit safeguards for publicly retrievable held-out measurements.\n\n**Completeness Check:**\n\n- Instruction: Complete — Defines the shared calibration parameters, required deliverables, prediction interval, and prohibited corrections.\n- Inputs: Complete — Numeric arrays contain 60 public dynamic frames, 420 quasi-static frames, and complete 500 Hz controls; dimensions, units, frame IDs, and alignment limits are documented.\n- Toolchain: Complete — Pinned dependencies, canonical simulator, and a provisioning template provide a reproducible workflow; runtime compatibility and performance were not independently verified.\n- Reference output: Complete — Required artifact names and trajectory schemas are defined. Separate reference arrays contain exactly the 70 withheld dynamic frames, IDs 60–129.\n- Rubric: Complete — Supplied scoring code matches the deterministic surface sampling, equally weighted frame statistics, thresholds, and 60/40 credit formula.\n\n**File and Task Reachability:**\n\nAttachments: `partially_reachable`, solely because the optional video preview was inspected structurally but not visually decoded. Task: `reachable` through the documented provisioned, offline workflow.\n\nInspected all six attachments: `agent_input.zip`, `Dockerfile`, `PREREQUISITES.md`, `verify_container.py`, `score.py`, and `ground_truth.npz`. The input archive’s 14 members were inspected, including documentation, scripts, JSON contracts, numeric containers, preview metadata, and checksum list. Downloaded sizes match the manifest; archive integrity checks passed, with no unsafe paths, encrypted entries, or excessive expansion. Numeric arrays have consistent schemas and no nonfinite values. Held-out frames are absent from the public input package.\n\nThe completion path is calibration against public observations, generation of the required artifacts, and a trusted canonical rerun against isolated references. Uploaded code was not executed. Simulation runtime, attainable accuracy, actual isolation enforcement, and external provenance were not independently verified.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\n`public_benchmark` — The submission explicitly identifies the source benchmark and states that withheld recordings are publicly retrievable. Mandatory offline execution, evaluator-asset isolation, and exclusion of external retrieval tools address runtime answer retrieval. No affirmative evidence establishes a memorized canonical fitted solution; external public facts were not independently checked.\n\n**Recommended Changes:**\n\nThe data split, fixed physics contract, scoring implementation, and isolation requirements are well specified. No contributor changes are required. Before reporting any candidate score, the evaluator must enforce and verify the documented isolation requirements; the supplied template alone does not certify a valid evaluation environment.","reviewed_at":"2026-10-11T03:34:35.913669Z"},"claim":{"id":"29685ccf-5a2e-407d-a8b2-8f555c7940c4","submission_id":"6fa6823d-49e4-46cd-bac1-fc3bc6b13434","submission_revision_id":"24dcc140-ea77-4d44-a2a2-eca694b250b0","builder":"builder-01","status":"active","claimed_at":"2026-10-15T16:20:51.918586Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}