{"claim":{"id":"a7c77860-f4e0-45ff-af45-cf006cb31a1c","submission_id":"71847f1e-b67c-4b16-b919-c39f9a23f322","submission_revision_id":"226c8d0c-829e-436b-9da2-2ddd42bf6198","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:08:22.863886Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"71847f1e-b67c-4b16-b919-c39f9a23f322","submission_revision_id":"226c8d0c-829e-436b-9da2-2ddd42bf6198","revision_version":2,"submission_status":"under_review","submitted_at":"2026-08-19T16:55:29.084271Z","submitter_name":"Abderrahim Bendahi","submitter_email":"abderrahimbendahi@gmail.com","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"gpt-5.6-sol","harness":"Codex","estimated_score":"40–60%"}},"submitted_at":"2026-08-19T16:55:29.084271+00:00","industry_domain":"Robotics / Sim-to-Real Transfer / Reinforcement Learning","task_short_name":"sim2real_partial_observation_calibration_instance_1","operating_system":"Linux","task_description":"Objective: Calibrate the joint physics, actuator-degradation, and sensing distribution of a partially observed MuJoCo robotic pushing system from masked offline trajectories, then select the safest of five opaque stateful PyTorch controllers.\n\nStarting State: The input package contains a nominal torque-controlled two-joint MuJoCo pushing simulator; 48 training trajectories and 18 public validation trajectories; five frozen controller checkpoints; physical parameter bounds; and loading and deployment-evaluation utilities. Ten latent quantities vary jointly by episode: object mass, surface friction, joint damping, actuator strength, actuator dead-zone, control latency, thermal decay, sensor latency, and two joint-encoder biases. Observations contain joint encoders and body positions but no velocities. Measurements may be missing and are accompanied by binary availability masks. The latent distribution may be multimodal and correlated.\n\nInstructions & Rules: Recover a physically valid joint parameter distribution while preserving episode ordering and action history. Distinguish control delay, sensor delay, and time-varying actuator degradation. Reconstruct latent motion and validate with genuinely independent, history-aware one-step predictions: initialize each transition separately from a filtered latent-state estimate and do not count prefixes of a from-reset rollout as one-step predictions. Separately perform at least 28-step open-loop prediction. Evaluate all five checkpoints under at least 384 fixed-seed samples drawn jointly from the fitted distribution. Rank controllers by 0.10 * expected_return + 0.90 * cvar_10, where cvar_10 is the mean of the worst 10% of returns and larger is better. Produce the five exact artifacts required by README.md: calibration.json, validation_metrics.json, controller_scores.csv, selected_controller.json, and calibration.py. calibration.py must reproduce all five artifacts through the documented task-root and output-dir CLI. Do not modify public data, simulator code or XML, bounds, or checkpoints.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Unpack sim2real_hard_v3_task_inputs_deepreview_fixed.zip and sim2real_hard_v3_reference_deepreview_fixed.zip so that sim2real_task/ and reference/ are sibling directories. Install the pinned open-source dependencies from sim2real_task/requirements.txt. Run:\n\npython reference/evaluate.py --task-root sim2real_task --agent-output <directory-containing-the-five-agent-artifacts> --reference-root reference --report evaluation_report.json\n\nThe evaluator SHA-256 verifies every public input and assigns a fatal zero for modified or missing public inputs, missing required outputs, or invalid calibration and validation schemas. It awards 35 points for recovery of the ten-dimensional joint distribution using normalized moments, multivariate energy distance, joint control/sensor-delay frequency, and severe-regime coverage; 10 points for genuinely independent hidden one-step transitions initialized separately with the necessary control-delay history; 15 points for a separate hidden 30-step from-reset open-loop rollout; 35 points for deterministic tail-robust performance of the selected controller across 1,024 hidden deployment episodes under 0.10 * expected_return + 0.90 * cvar_10; and 5 points for the exact five unique controller rows, finite arithmetic, exact criterion string, minimum sample counts, exact validation-metric fields, correct reported selection, and controlled reproduction of all artifacts by executing calibration.py in a temporary read-only public-task copy. Distribution and prediction components are normalized to the included genuine public-input-only expert run. A clean extraction of the replacement archives gives the expert 100/100 with no evaluator notes. All hidden checks and rollout seeds are deterministic.","software_and_version":"Python 3.12; MuJoCo 3.11.0; Gymnasium 1.3.0; PyTorch 2.13.0; NumPy 2.5.2; SciPy 1.18.0","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"fd3c7096-dde1-405b-b07e-6b71a10e0d77","filename":"01_INPUT_MATERIALS__INSTANCE_1.zip","category":"input_materials","content_type":"application/zip","size_bytes":423775},{"file_id":"c71ccb0e-08f9-473c-90f0-0435b63c0958","filename":"02_REFERENCE_OUTPUT__INSTANCE_1.zip","category":"reference_output","content_type":"application/zip","size_bytes":443303}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — the submission is complete, coherent, objectively evaluable, and end-to-end reachable, with only a negligible artifact-count wording inconsistency.\n\n**Completeness Check:**\n- Instruction: Complete — clearly defines the ten latent parameters, partial-observation constraints, validation protocols, controller ranking criterion, and five required artifacts.\n- Inputs: Complete — includes 48 training and 18 validation episodes, parameter bounds, a functioning simulator specification, loaders, evaluation utilities, and five distinct controller checkpoints.\n- Toolchain: Complete — pins the required open-source Python, MuJoCo, Gymnasium, PyTorch, NumPy, and SciPy stack for Linux.\n- Reference output: Complete — provides evaluator-only ground truth, hidden trajectories, deployment scores, expert artifacts, and a structurally valid 100/100 expert report.\n- Rubric: Complete — the evaluator implements the stated 35/10/15/35/5 scoring allocation, fatal integrity/schema checks, independent history-aware one-step evaluation, open-loop evaluation, and deterministic controller selection.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. I inspected `01_INPUT_MATERIALS__INSTANCE_1.zip` and `02_REFERENCE_OUTPUT__INSTANCE_1.zip`, including all contained files: documentation, source and XML, JSON and CSV schemas, NumPy archives and arrays, PyTorch checkpoint containers, hidden evaluator assets, and expert outputs. Both archives match their manifest sizes, are readable and unencrypted, pass container integrity checks, have modest expansion, and contain no traversal paths, links, devices, or unexpected executables. Numeric arrays have consistent shapes, finite values, contiguous episode timesteps, valid binary masks, and documented measurement ordering; all 19 public inputs match the supplied authoritative hashes. Reference and hidden assets are separated from solver-visible inputs.\n\nNo archive members were left unreviewed. Uploaded programs were statically inspected and not executed, as required. The review environment lacked a Python runtime, so the supplied expert run was validated structurally rather than reproduced; exact external package availability was not independently searched. The documented completion path is to install the pinned dependencies, run the submitted calibration CLI against the public task root to generate the five artifacts, and invoke the separate evaluator against those artifacts and the evaluator-only reference directory.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. External public-identifiability searches were not permitted, so this classification relies on the submitted and locally inspectable evidence.\n\n**Recommended Changes:**\nThe task is professionally scoped and ready; no material changes are required. Optionally clarify whether diagnostic files are permitted, because the reference script writes `per_episode_estimates.csv` in addition to the five stated artifacts. As defense in depth, execute submitted `calibration.py` inside a network-disabled OS sandbox exposing only the read-only public task and writable output directory.","reviewed_at":"2026-08-19T17:04:35.551717Z"},"claim":{"id":"a7c77860-f4e0-45ff-af45-cf006cb31a1c","submission_id":"71847f1e-b67c-4b16-b919-c39f9a23f322","submission_revision_id":"226c8d0c-829e-436b-9da2-2ddd42bf6198","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:08:22.863886Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}