{"submission_id":"20b19386-c3c2-476f-8112-3ef1184dfbb3","submission_revision_id":"9139b638-1eb5-4d5a-ad20-8f53228144bc","revision_version":4,"submission_status":"under_review","submitted_at":"2026-10-09T02:27:11.560638Z","submitter_name":"Jixian Liu","submitter_email":"jliu376@jh.edu","task":{"status":"submitted","version":4,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5.5","harness":"Claude Code","estimated_score":"20–40%"}},"submitted_at":"2026-10-09T02:27:11.560638+00:00","industry_domain":"Mechanical Engineering, Robotics","task_short_name":"ur5e_fr3_motion_planning_stack","operating_system":"Linux","task_description":"Objective: Deliver a reproducible motion-planning stack for the two manipulators of a lab cell, a Universal Robots UR5e (6 joints) and a Franka Robotics FR3 (7 joints, with hand), on an offline lab machine. The robots of the cell are emulated: each is a hidden, high-fidelity MuJoCo hardware emulation (synthetic plant) whose true parameters and effects differ from the published description, standing in for the physical robot. The work has four stages that build on each other: a verified ROS 2 / MoveIt bring-up; MuJoCo models and one shared torque controller that must hold and track on the two emulated robots, which have to be identified from hardware logs recorded on them; scene-consistent collision checking with one shared A* and one shared RRT; and a shared dual-arm cell in which both robots are planned and executed together.\n\nStarting State: An offline Ubuntu 22.04 VM with ROS 2 Humble, MoveIt 2 2.5.9, ros2_control and the build tools installed; the agent has sudo. A pre-existing workspace of another project is restored to ~/ros2_ws and ~/.bashrc sources it; it contains a modified fork of ur_description. input/ (read-only, 454 MB):\n- TASK.md, the full brief: every deliverable and the exact command-line and Python interfaces the evaluator calls. It names what is examined but not the checks or tolerances.\n- schemas/ (17 JSON schemas), examples/, and cell/cell_example.json (an example dual-arm cell).\n- offline_apt/: a flat APT repository (UR stack, Franka build dependencies, apt libfranka 0.20.4, and stale franka_description 1.0.1 / franka_msgs 1.0.0 that must not be used).\n- sources/: bare git mirrors of franka_ros2, franka_description, libfranka and libfranka-common with the pins.\n- runtime_env/: hashed wheel lock of the task Python runtime (mujoco 3.14.0, numpy, scipy, ...); software/make_python_env.sh builds it offline.\n- hardware_logs/<robot>/: recordings made on the cell's emulated robots (deterministic synthetic data from the hidden MuJoCo plants) with the lab's PD + published-gravity controller at 500 Hz (static poses, single-joint sweeps at six speeds, torque steps while sliding, six excitation trajectories, two validation trajectories) and a DATASHEET. The emulated robots differ from the published descriptions: an unmeasured end-effector load, joint friction including breakaway stiction, rotor inertia, deviating link inertias, drive torque constants, a transport delay plus drive dynamics, torque ripple, and noisy quantized encoders with a filtered velocity estimate.\n\nInstructions & Rules:\n- Stage 1: install only the missing robot packages from the offline repository (check names first; no upgrades, no wildcards, no removals); build the pinned Franka sources in a colcon workspace inside output/ (a submodule must be redirected to the local mirror); write setup_env.sh so that a new interactive terminal resolves the official UR packages and the pinned Franka packages and the old workspace does not shadow them; provide bin/bringup_moveit for both robots (headless move_group on mock hardware) and a machine-readable manifest; verify everything with live queries.\n- Stage 2: MJCF models of both robots from the official descriptions (torque motors, per-parameter provenance, FK checked against MoveIt); ONE computed-torque controller for any number of joints behind api/controller_api.py; thresholds declared and committed before the validation runs. Identify both emulated robots from the hardware logs (payload mass/COM/inertia, Coulomb and viscous friction, armature, torque scale, transport delay), deliver identified MJCFs and identification files, and configure the controller so that it holds and tracks on the emulated robots at 500 Hz from measured states.\n- Stage 3: one scene file loaded identically into MuJoCo and MoveIt (bin/mujoco_build_scene, bin/moveit_apply_scene); full-robot collision checking with edge subdivision (bin/check_states) and five regression cases; ONE lattice A* and ONE RRT for any n (api/planner_api.py, bin/plan); a benchmark whose settings are committed before its results.\n- Stage 4: the shared cell — one combined robot description (UR5e names prefixed ur_), bin/bringup_cell with mock hardware moving both arms and ONE move_group (groups ur_manipulator, fr3_arm, both_arms; the cell's obstacles and the UR tool), and bin/mujoco_build_cell, bin/check_cell_states (arm-against-arm checking) and bin/cell_plan (coordinated 13-joint planning within a time budget).\n- Everything goes into output/, which must be a git repository. Time: 2 hours from the start of the task to the last commit; every minute beyond costs 2 points of 100. Every check earns its weight when correct and costs it when wrong or missing (negative marking). The stage weights are Stage 1 5%, Stage 2 60%, Stage 3 15%, Stage 4 20%.\n\nThe evaluator re-runs every deliverable against hidden inputs, including the two hidden emulated robots, so reports are not accepted as evidence.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The grading runs on the VM after the agent has finished. It is deterministic except for the wall-clock-limited planners.\n\nSetup and sealing. The evaluator (root) stages hidden data (about 86 MB) and seals it, its own code and its work files into root-only directories. It runs every submission entry point (setup_env.sh, bin/*, api/ modules) under an unprivileged identity with no new privileges, no access to the hidden data, write access only to the paths it hands over, and no network for the non-ROS entry points. The planner API's collision checker is an out-of-process service, so the obstacle set never enters the submission's process. Hidden data:\n- reference kinematics and dynamics of the official descriptions; hidden scenes with non-identity base transforms; 264 labeled MuJoCo states + 48 edges and labeled MoveIt states (exact-mesh FCL labels with robustness margins);\n- 19 analytic planner-API problems (n = 2..9, known A* optima) and 16 robot planning problems;\n- two synthetic MuJoCo models (n = 3 and 9) for the control law;\n- the two true plants (the emulated robots: MuJoCo models with secret parameters) with held-out recordings, and per robot 3 holds, 3 tracks, 2 fast tracks and 4 aggressive tracks;\n- three hidden dual-arm cells (facing, corner, and a wall with a slot), each with 8 FK configurations, 44–54 labeled states (free, near-miss, arm-against-arm, arm-against-obstacle, self-collision), 7–10 labeled edges and 3 coordination problems with budgets of 20–105 s.\n\nScore = 0.05·S1 + 0.60·S2 + 0.15·S3 + 0.20·S4 − 0.02 per minute beyond 2 h (start() to the last commit), clamped to [0, 1]. Within a stage, a check with credit p contributes w·(p − (1 − p)), and the stage score is clamped to [0, 1].\n\n- S1: setup_env.sh from a new interactive shell (official UR packages, pinned Franka packages, ROS_DOMAIN_ID kept); the manifest against dpkg, package.xml and git; for each robot the live move_group of bringup_moveit (group, joints, joint limits against the official descriptions, /compute_fk at hidden configurations to 1e-6, /joint_states, plan-and-execute, clean shutdown); read-only inputs, the old workspace, the shell startup files and the apt rules untouched.\n- S2: models (joint ranges, motors with capacity = effort, EE FK to 1e-5, M(q) and g(q) against the reference to 1e-3, provenance); the exact control law to 1e-6 including saturation, also on the unseen synthetic models; evidence (metrics reproduce from the logs, re-simulation within 1e-6 rad, thresholds committed first). Identification against the true plant: transport delay exact; payload mass / COM within 6% / 10 mm (UR5e) and 11% / 36 mm (FR3); Coulomb / viscous friction within 13% / 32% (UR5e) and 10% / 15% (FR3) on every joint; held-out torque prediction NRMSE per joint within 1.5× that of the true rigid-body model. Closed loop owned by the evaluator, on the TRUE plant with the submitted controller and identified model, from measured (noisy, quantized, delayed) states: per hold and per track an error band of 2× the reference solution's own error on that case (holds 0.085–0.33 mrad after 2 s; track RMS 0.26–1.27 mrad, with max and final limits); aggressive tracks near the velocity and torque limits scored continuously, credit min(1, (target / RMS)²) with the target at the reference's level (0.20–0.25 mrad FR3, 0.82–0.84 mrad UR5e). A run that goes unstable, touches anything or keeps a drive saturated scores 0 on all closed-loop checks of that robot.\n- S3: scene loaders in MuJoCo and MoveIt on the hidden scenes (1e-6, attached tool); check_states and MoveIt /check_state_validity on the labels (100% in every category); planner API (A* cost = lattice optimum to 1e-6, statuses, RRT validity and seed determinism); bin/plan on the hidden robot problems with every path re-validated with exact meshes at 5 mm; the benchmark (scenarios, 10 seeds, valid paths, frozen settings committed first).\n- S4: for each hidden cell, the live cell of bringup_cell (model and groups, FK of both tips to 1e-5, planning scene, /check_state_validity on the labeled states at 100%, simultaneous plan-and-execute of both arms to 1e-3 rad, shutdown); the MuJoCo cell model; check_cell_states on states and edges (100%); cell_plan on the coordination problems — the path must start and end exactly at the given states and pass an exact-mesh re-validation at 5 mm on the true geometry, credit min(1, reference cost / path cost) when it arrives within the budget. Planning credit counts only if the MuJoCo collision checking is correct in every category.\n\nProvenance of the Stage 2 data. No physical robot was recorded. The two robots are emulated: each true plant is the official MuJoCo description plus secret, sampled deviations (payload mass/COM/inertia, link inertias, rotor inertia, Coulomb and viscous friction, torque constants) and effects outside the rigid-body model (Stribeck stiction, drive torque ripple, transport delay plus first-order drive lag, encoder quantization and noise, a filtered velocity estimate), stepped at 2 kHz under a 500 Hz control loop. The hardware logs and the held-out recordings are deterministic synthetic data generated from these plants with the documented lab controller (build/make_real_cell.py in the reference archive); the brief and the datasheets say so. The identification references are the plants' true parameters, and the closed-loop references come from running controllers on the same plants.\n\nHow the ranges were set (narrow, reference-anchored). The hidden test data come from the true plants and exact-mesh checkers. Each Stage 2 tolerance is 2× the error the reference solution actually achieves on that case (with floors above sensor noise; delays exact). Collision labels are deterministic, so 100% is required. Every coordination problem was screened inside the grading container: the reference planner, called as the grader calls it, must solve it 5 times out of 5 within its budget, and the cost bar is 1.05× the median of those 5 runs.\n\nWhy it is hard. Stage 2 carries 60%: the identification has to be right on robots the agent cannot run (the transport delay is ambiguous with the drive dynamics; the small FR3 payload is biased by samples inside the stiction regime), and the controller has to perform on the true plants within narrow, reference-anchored bands.\n\nReference result. Validated in one clean end-to-end run under the final rules: a fresh offline VM container, start(), the reference solution (33 minutes to its last commit), then the official evaluate(). Score 1.000: 166 of 166 checks, no evaluator errors, no time penalty. Identification against the true plants: UR5e payload 3.0% / 1.5 mm, FR3 5.6% / 17.9 mm, both delays exact; every hold and track within its band; all 9 coordination problems solved within budget and below the cost bar. The identification and closed loop are deterministic, so the clean run reproduces the build-time numbers from which the bands were derived.\n\nSelf-test. A fresh-context Claude Opus 5.5 agent in Claude Code, given only the task prompt and the offline VM, finished in 76 minutes (no time penalty; it ran three helper agents in parallel). The official evaluate() under the final rules scores it 0.361: S1 0.81, S2 0.00, S3 0.80, S4 1.00, 147 of 166 checks passed. It built a complete, working stack — MoveIt bring-up, models, the exact control law, collision checking that agrees with MoveIt, a dual-arm cell whose coordinated planner returned paths below the reference cost bar on all 9 hidden problems — but it identified both drive delays wrongly (4 and 3 ticks; true 2 and 1) and both payloads outside the range (UR5e 14%, FR3 18%), so its controller fails every hold and track on the emulated robots and reaches only 0.13–0.23 credit on the aggressive tracks; it also used ±2π instead of the official ±6.1332 rad UR5e joint limits, which costs Stage 1 checks and makes its planner reject the hidden UR5e problems. Earlier task versions without the hidden emulated robots and narrow ranges were solved at 0.94–0.99 (see the evidence). The run was graded after the stage weights and hidden ranges were finalised; its deliverables and the brief's interfaces are unchanged (details in the evidence NOTE).","software_and_version":"ROS 2 Humble on Ubuntu 22.04 (apt snapshot 2026-08-07), MoveIt 2 2.5.9, ros2_control, Universal Robots ROS 2 packages (offline APT), franka_ros2 v2.7.1 / franka_description 2.8.1 / libfranka 0.20.4 (pinned sources), MuJoCo 3.14.0, Python 3.10 (NumPy, SciPy)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"a56e420f-05d9-461c-96c1-dd0c1d60269b","filename":"ur5e_fr3_motion_planning_stack_selftest_evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/x-zip-compressed","size_bytes":32570},{"file_id":"422b3a4d-ab97-43f5-9c0e-1c39ae18f604","filename":"ur5e_fr3_motion_planning_stack_reference.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":206452330},{"file_id":"a3c0bfbc-e90f-4f44-ae22-6e3efab0d3ac","filename":"ur5e_fr3_motion_planning_stack_inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":473417788}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — Revision 4 provides a coherent, reproducible robotics task with complete inputs, reference implementation sources, and objective evaluation.\n\n**Completeness Check:**\n\n- Instruction: Complete — The full brief defines all four stages, deliverables, interfaces, operating constraints, and scoring.\n- Inputs: Complete — Offline packages, pinned source mirrors, 17 schemas, examples, and documented synthetic recordings are supplied; all 371 input/software checksums match.\n- Toolchain: Complete — Ubuntu 22.04, ROS 2 Humble, MoveIt, and the hashed Python runtime have a documented setup path with offline task dependencies.\n- Reference output: Complete — Reference sources and reproduction recipes cover every stage; the supplied clean-run report records 166/166 passing checks and a 1.000 score.\n- Rubric: Complete — Grading code implements hidden-input verification, continuous credit, negative marking, stage weights, and the time penalty.\n\n**File and Task Reachability:**\n\nAttachment classification: `reachable`. Inspected all three attachments: `ur5e_fr3_motion_planning_stack_selftest_evidence.zip`, `ur5e_fr3_motion_planning_stack_reference.zip`, and `ur5e_fr3_motion_planning_stack_inputs.zip`. Their byte counts match the manifest, ZIP integrity checks pass, and no encryption or unsafe top-level paths were found. Inspection covered archive inventories, schemas, representative source content, dependency metadata, numeric recording structures, hidden fixtures, and evaluation reports. Individual dependency implementations were not exhaustively reviewed.\n\nTask classification: `reachable`. One suitable machine can build the offline stack, produce the specified artifacts, and undergo objective evaluation against separately staged hidden data. Solver-visible recordings exclude the true-state channels present in held-out evidence. The robotics workflow was not executed during this review, and public availability was not independently verified. Supplied runs document a network-namespace limitation within an offline test environment.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\nNone. The recordings are explicitly disclosed as deterministic simulation data; this does not establish AI fabrication or a synthetic defect fixture. Public-benchmark status remains unverified.\n\n**Recommended Changes:**\n\nThe submission provides substantial reference and grading evidence and transparently distinguishes the earlier self-test from its final-rule regrading. No changes are required.","reviewed_at":"2026-10-09T02:47:14.617393Z"},"claim":null}