{"claim":{"id":"6ba4447e-6d7f-43db-ab20-82304eee470e","submission_id":"ec815f4e-cc39-40af-b49e-8f347d5aba02","submission_revision_id":"f9db4446-7061-41e9-b974-020d347556a3","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:33:13.380280Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"ec815f4e-cc39-40af-b49e-8f347d5aba02","submission_revision_id":"f9db4446-7061-41e9-b974-020d347556a3","revision_version":2,"submission_status":"under_review","submitted_at":"2026-09-29T12:38:52.686066Z","submitter_name":"Jixian Liu","submitter_email":"jliu376@jh.edu","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"10–20%"}},"submitted_at":"2026-09-29T12:38:52.686066+00:00","industry_domain":"Robotics","task_short_name":"campus_robot_localization_1","operating_system":"Linux","task_description":"A university runs a two-wheeled, self-balancing research robot (Segway RMP base: wheel speeds, a fibre-optic gyro, an IMU, a 2-D lidar) on its campus in every season. The recordings are real: excerpts of the University of Michigan North Campus Long-Term (NCLT) dataset, converted to ROS 2 bags without changing any sensor value or timestamp.\n\nThe robot software is the classic stack, set up with defaults: differential-drive wheel odometry with a guessed track width, slam_toolbox mapping, and Navigation2 map_server + AMCL localization started at the operator's start pose. On these recordings it fails at every level:\n- the wheel-derived heading is useless (the balancing base's effective track is not the guessed one);\n- the wheel speeds arrive in bursts and their scale drifts 5-8% between sessions months apart;\n- the lidar sees the ground when the body pitches, and its own chassis/mast at the edges of the field of view;\n- slam_toolbox cannot map the outdoor routes (its scan matcher applies corrections regardless of match quality, and nothing anchors the heading), so the saved map folds;\n- AMCL's default motion model spins the heading away outdoors, and the operator's start pose is ~0.36 m / 4 deg off the true start.\n\nThe agent gets three development sessions of one part of the campus with survey-grade ground truth (a mapping session, and two operation sessions five weeks and two months later). It must repair and complete the software so that ONE build maps an area in one session and then localizes the robot on that map in later sessions, starting from the operator's start pose, with no per-session constants and no place-specific knowledge.\n\nOnly two ROS packages (robot_base, robot_bringup) may be changed. The evaluator then replays hidden recordings of ANOTHER part of the campus, recorded in winter: a mapping session, then two operation sessions one and three weeks away, through the submitted launch files at 1x speed. Each replay takes 3-12 minutes of wall time, so a claimed fix must be demonstrated, not just argued.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The grading is deterministic apart from replay timing. It runs on the VM as root after the agent has finished:\n\n1. Verify with dpkg that the ROS/Python/shell system packages were not modified.\n2. Build campus_msgs from the evaluator's own copy and the submitted robot_base/robot_bringup packages as an unprivileged user, offline.\n3. Replay the hidden mapping session through `ros2 launch robot_bringup mapping.launch.py`, then the two hidden operation sessions one after another through `ros2 launch robot_bringup localization.launch.py` on the map just produced (each started at its operator start pose, placed in the map frame with the mapping run's own trajectory-to-ground-truth alignment). All replays run at 1x in real time with a simulated clock; the runner audits that only the declared TF edges and topics are published and that every sensor-consuming node uses sim time.\n4. Score, per session: replay validity and coverage; odometry heading/distance drift over 10 m windows (<=4.0 deg / <=6.0%, at most 5 jumps); for the mapping session, SLAM trajectory RMSE/max after SE(2) alignment (<=2.0 m / <=6.5 m) and map precision/completeness against a reference occupancy map built from the ground truth (>=0.65 / >=0.40, within 0.25 m); for the operation sessions, localization position RMSE/max in the held-out map's frame, ignoring the first 15 s (<=2.5 m / <=5.0 m).\n\nScore = 0.05*S1 (submission contract) + 0.10*S2 (development evidence reproduces) + 0.85*S3 (held-out re-execution above), where S3 = 0.4*(weighted pass fraction) + 0.6*(all held-out checks pass) — one build has to work everywhere, not just on average.\n\nThresholds were calibrated by replaying the reference solution on the development area and on a separate calibration area (not used elsewhere), leaving ~1.6-2x margin over the worse of the two; the held-out area was replayed afterwards only as a check.","software_and_version":"ROS 2 Humble (Ubuntu 22.04): Cartographer (cartographer_ros 2.0.9002), slam_toolbox 2.6.10, Navigation2 1.1.20 (map_server, AMCL), rosbag2","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"7ad07202-1ee3-41e9-bd88-e10f40974d6a","filename":"inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":39183761},{"file_id":"a6c7159a-b1a5-4ddb-b6bd-05f4e3019b55","filename":"reference.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":95700180}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, professionally scoped, internally verifiable, and executable end to end, with only a negligible documentation inconsistency.\n\n**Completeness Check:**\n- Instruction: Complete — The editable scope, ROS interfaces, launch contracts, prohibited behavior, deliverables, and expected end state are explicit.\n- Inputs: Complete — Three development ROS 2 bags, corresponding 20 Hz survey-grade trajectories, configs, manifests, examples, starter packages, and platform documentation are present and consistent.\n- Toolchain: Complete — Ubuntu 22.04, ROS 2 Humble package versions, installed dependencies, offline constraints, hardware expectations, and Apache-2.0/ODbL licensing are specified.\n- Reference output: Complete — Protected evidence includes development expectations, matching truth copies, a calibrated protocol, three held-out datasets and trajectories, and a usable 0.1 m reference-map archive.\n- Rubric: Complete — Stage weights, per-check weights, validity rules, trajectory alignment, timing tolerances, map criteria, thresholds, and all-pass behavior are objective and mutually consistent.\n\n**File and Task Reachability:**\nThe attachment classification is `reachable`, and the task classification is `reachable`. Both `inputs.zip` and `reference.zip` were inspected. They contain 52 and 28 regular files respectively, are unencrypted, pass container integrity checks, match their declared byte sizes, and contain no traversal paths, links, devices, or unsafe expansion. All six bag manifests match their files by size and SHA-256; the SQLite databases pass integrity checks and contain the seven declared CDR topics with matching message counts and plausible representative records. All JSON parses correctly, all six truth trajectories have the documented schema, finite values, and strictly increasing timestamps, and protected development expectations match the supplied configs, manifests, and scan counts. The reference map contains five non-object NumPy arrays with consistent 2,511 by 2,141 dimensions, finite distance fields, 26,935 occupied cells, 0.1 m resolution, and a valid origin. Frozen-project hashes and duplicated development truth also match.\n\nNo manifest attachment was left unreviewed. Representative sensor records were inspected rather than exhaustively replaying every message; no material tooling limitation remains. Solver-visible development material and protected held-out evaluation material are separated into distinct attachments. The end-to-end path is clear: modify only the two permitted packages, build offline against ROS 2 Humble, replay the development mapping and localization sessions, submit the specified source and run artifacts, then undergo an unprivileged rebuild and sequential hidden mapping/localization replay with deterministic trajectory and map scoring on one ordinary 8-vCPU machine.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The submission explicitly identifies its recordings as excerpts of the publicly named NCLT dataset. The task itself, package repair, evaluation protocol, hidden area selection, and expected solution are custom; solver execution is offline, and held-out recordings, truth, and reference map are protected, substantially mitigating answer retrieval.\n\n**Recommended Changes:**\nThe submission is ready and no material changes are required. As optional cleanup, change the `FogHeading` message comment from “unwrapped” to “wrapped to [0, 2π)” so it agrees with the robot documentation and observed wrap transitions. Documenting the exact reference-map precision and completeness formulas would also improve transparency without changing task reachability.","reviewed_at":"2026-09-29T12:49:58.449547Z"},"claim":{"id":"6ba4447e-6d7f-43db-ab20-82304eee470e","submission_id":"ec815f4e-cc39-40af-b49e-8f347d5aba02","submission_revision_id":"f9db4446-7061-41e9-b974-020d347556a3","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:33:13.380280Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}