{"claim":{"id":"649e93aa-c437-4cd5-a7ba-cf7e53f95e89","submission_id":"32ec6498-6ff8-4b4f-a934-c11907f955c1","submission_revision_id":"87710418-b7a3-4428-b733-743d5fa49f67","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:40:03.383820Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"32ec6498-6ff8-4b4f-a934-c11907f955c1","submission_revision_id":"87710418-b7a3-4428-b733-743d5fa49f67","revision_version":3,"submission_status":"under_review","submitted_at":"2026-10-08T19:11:11.168876Z","submitter_name":"Jixian Liu","submitter_email":"jliu376@jh.edu","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5.5","harness":"Claude Code","estimated_score":"20–40%"}},"submitted_at":"2026-10-08T19:11:11.168876+00:00","industry_domain":"Robotics","task_short_name":"slam_sensor_degradation","operating_system":"Linux","task_description":"Objective: Make the SLAM stack of a small indoor robot work again after the robot is re-deployed in a worse physical configuration. Configure and compare Cartographer and RTAB-Map (lidar only) on a TurtleBot3 Burger with a 2-D lidar and wheel odometry in a simulated three-room house (AWS RoboMaker Small House, Gazebo Classic 11), select ONE package, and deliver a reproducible configuration that also holds on a route it has never seen.\n\nStarting State: An offline Ubuntu 22.04 VM with ROS 2 Humble, Cartographer, RTAB-Map, Nav2 and Gazebo Classic 11 installed; the agent has sudo.\n- input/ (read-only, 64 MB): TASK.md (the full brief), task_contract.json, format examples, and four rosbag2 recordings of the same commanded 16-waypoint development route (about 330 s, 45 m), each with its own simulator ground truth. The recordings were made in Gazebo with physically modified robots, not synthesised:\n  - baseline: the standard robot;\n  - occluded: a payload rack mounted 0.13 m behind the lidar and a cheaper 81-beam lidar unit; the rear 120 degrees (27 beams) return the rack itself in every scan, so the room is seen only through the forward 240 degrees;\n  - slip: a slippery floor; the drive wheels slip in turns and in the steering corrections of straight driving, so the wheel-encoder odometry drifts, mostly in heading, while it still reports its usual covariance;\n  - combined: both.\n- software/project/ (122 MB, copied to output/source/ at start): the ROS project with a runner that replays a bag at 1x through either SLAM package and captures the ONLINE map->base_footprint pose at every scan stamp and the final map, the public scorer, the two editable SLAM packages, the TurtleBot3 models including the three degraded variants, the house world and documentation.\n\nInstructions & Rules:\n- Build the workspace offline and run the 2 x 4 matrix (algorithm x condition), scoring every run against the ground truth of its own recording with the project's scorer.\n- Diagnose why each package degrades: rack returns that are mapped and matched as an obstacle moving with the robot unless filtered; scan matching with only 54 usable beams; odometry whose heading drifts with the slip; and the captured pose being the SLAM's map->odom composed with the RECORDED odom->base_footprint at each scan stamp.\n- Change only turtlebot3_cartographer/ and turtlebot3_rtab/: parameters, launch files, condition-specific configurations selected through the ALE_EXPERIMENT_CONDITION variable, and adapter nodes (filters, odometry estimators, TF composers). The map must still come from the compared package. One SLAM TF publisher, only the map->odom edge, simulation time, no prior maps, no other SLAM/localisation back end, no data-dependent tricks beyond the public condition name.\n- Select ONE package, freeze the source, and re-run all eight cells with it.\n\nDeliverables (output/): source/ (the complete project); runs/<algorithm>/<condition>/ (the eight final runs written by benchmark.cli run); metrics.json (development RMSE and coverage from the scorer); analysis.md (sections Configuration, Results and Selection, and a \"Selected algorithm:\" line); submission.json (index of all of these).\n\nThe evaluator then replays hidden recordings of a DIFFERENT route through the same house, made with the same four robots, plus one supplemental cell (the combined recording with unpublished synthetic odometry jitter), through the delivered configuration. The RMSE limits are hidden, so the agent must engineer robustness rather than tune to a number. The task is not \"run a command\": on the development recordings the delivered configurations reach Cartographer 1.7 / 233 / 19 / 151 cm and RTAB-Map 0.9 / 39 / 65 / 30 cm (baseline / occluded / slip / combined), far from what the evaluator requires. Each replay costs about 6 minutes of wall time.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The grading runs on the VM after the agent has finished and is deterministic apart from the replay timing.\n\n**Setup.** The evaluator (root) stages hidden data, about 100 MB:\n- held-out recordings of another route through the house (about 370 s, 50 m), made with the same four robot variants (baseline, occluded, slip, combined), each with its own simulator ground truth, plus the supplemental combined_jitter cell (the combined recording with an unpublished per-sample odometry jitter draw);\n- a reference occupancy map built from the full-resolution held-out scans at the ground-truth poses;\n- the manifests, scan stamps and ground truth of the four development recordings, and the manifest of the frozen project;\n- the reference solution and the grader source (reference/evaluator/), for the maintainers.\n\n**Sealing and re-execution.** The evaluator seals the hidden data and its own code into root-only directories, verifies with dpkg that the ROS and Python packages were not modified (the agent had sudo), and stops leftover ROS processes. It rebuilds only the two submitted packages, offline, as a throwaway unprivileged user, then replays each held-out bag at 1x with the project's own replay and capture code, running as root. The submitted launch file runs as the unprivileged user, cannot read any bag or truth file, and receives the data only over DDS. The replay enforces the TF, clock and topic ownership rules and records which nodes publish /map; the evaluator also watches for other SLAM/localisation back ends.\n\n**Score** = 0.05·S1 + 0.05·S2 + 0.85·S3 + 0.05·S4. Each stage is the weighted credit of its checks, except S3.\n- S1, contract: valid index of the eight cells; output-tree rules; every file outside the two packages byte-identical to the frozen project; package rules (text files only, size limits, no prior maps); the offline, unprivileged build succeeds.\n- S2, development evidence: each of the eight runs completed on the right recording (checked against the evaluator's own manifests) with unchanged replay settings, was made by the frozen runner with the submitted packages, and has a valid trajectory sampled exactly at the scan stamps and a valid map; the reported metrics reproduce (1 mm + 1 %); the analysis has the required structure and a consistent selection.\n- S3, held-out replay of the selected package on five cells. The gated cells baseline, occluded and slip each have a run, an RMSE and a map check (weights 0.25 / 3 / 0.75). One configuration must meet the requirements in every gated condition, so S3 = 0.3 × (weighted credit of all S3 checks) + 0.7 × [every check of the gated cells fully passed]. The robustness cells combined (run / RMSE 0.25 / 3) and combined_jitter (0.1 / 1.2) count in the weighted credit only and have no map check, because with the occluded 81-beam lidar and slipping wheels together the heading is only weakly observable in places and even the reference varies between replays.\n  - run: the replay completes with a valid trajectory (coverage ≥ 0.95, gaps ≤ 0.5 s) and a valid map published by the compared package itself, with no other back end running;\n  - RMSE: position RMSE after SE(2) alignment against that recording's ground truth; credit 1 at or below the hidden limit L, 0 at or above 2L, linear between. L = 1.5 × the reference's held-out RMSE, rounded up to 1 mm: baseline 0.011 m, occluded 0.007 m, slip 0.033 m, combined 0.290 m (also used for combined_jitter);\n  - map (gated cells): precision ≥ 0.8 (occupied cells within 0.1 m of a real lidar return), completeness ≥ 0.4 (reference walls covered), and an A* tour from the start through four goals and back, planned on the submitted map only, that keeps ≥ 0.05 m from real obstacles.\n- S4, held-out replay of the other package on baseline, occluded and slip: each completes with a valid trajectory and map and RMSE ≤ 0.5 m.\n\n**Reference result.** The shipped reference (RTAB-Map, condition-aware) was validated in one clean end-to-end run under this final protocol: fresh VM container, start(), reference agent, evaluate(). Score 1.000 with no evaluator errors, all frozen files identical and all eight development trajectories valid. Held-out RMSE: baseline 0.73 cm, occluded 0.43 cm, slip 2.19 cm (map precision 0.98–1.00 in all three), combined 19.3 cm, combined_jitter 19.6 cm; Cartographer (S4) 2.5 / 15.1 / 21.2 cm. The limits were set from a separate calibration evaluate() with provisional limits and single-cell replays, which are listed in reference/protocol.json and are not claimed as validation.","software_and_version":"ROS 2 Humble on Ubuntu 22.04 (apt snapshot 2026-08-07): Cartographer 2.0.9002, RTAB-Map 0.23.7, Nav2 1.1.20, Gazebo Classic 11.10.2","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"95c73aea-04c2-4894-b7df-05bc0779ba39","filename":"slam_sensor_degradation_reference.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":15108124},{"file_id":"467ab5bf-5186-4fad-910a-bc5b2263ebab","filename":"slam_sensor_degradation_inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":97586373},{"file_id":"887ac5fc-4978-4d52-8a3a-406dbdf8746a","filename":"slam_sensor_degradation_selftest_evidence.zip","category":"difficulty_self_test_evidence","content_type":"application/x-zip-compressed","size_bytes":26694}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally coherent, objectively evaluable, and demonstrably reachable with a validated reference solution.\n\n**Completeness Check:**\n- Instruction: Complete; the objective, constraints, allowed changes, deliverables, and evaluation procedure are explicit.\n- Inputs: Complete; development recordings, manifests, ground truth, project files, and contracts are present and usable.\n- Toolchain: Complete; the specified ROS, Gazebo, SLAM packages, replay runner, and offline build path are available.\n- Reference output: Complete; the reference archive includes held-out data, evaluator materials, protocol information, and a validated solution.\n- Rubric: Complete; S1–S4 define reproducible artifact, trajectory, map, RMSE, robustness, and package-comparison checks.\n\n**File and Task Reachability:**\n\nAll three uploaded archives were inspected for inventory, formats, sizes, path safety, and encryption. No traversal paths, inaccessible required files, or unsafe archive entries were observed. The evidence includes a validated reference evaluation scoring 1.000, while the independent self-test candidate achieved partial credit and exposed the intended robustness difficulty. The task is therefore reachable without relying on undocumented access or contributor-only information.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor.  \nConfidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\n- `synthetic_defect_fixture`: Supported by the evidence. The task intentionally uses predetermined degraded robot variants and a supplemental odometry-jitter condition as controlled evaluation fixtures.\n- `ai_synthetic`: Not supported. Nothing inspected indicates that the core recordings, ground truth, or reference materials were AI-fabricated.\n- `public_benchmark`: Not established. Public identifiability of this complete task, dataset, and answer was not demonstrated.\n\nThe controlled degradation fixtures do not undermine the task’s validity; they are explicitly documented, physically motivated, and paired with hidden-route evaluation.\n\n**Recommended Changes:**\n\nNone.","reviewed_at":"2026-10-08T19:21:02.016859Z"},"claim":{"id":"649e93aa-c437-4cd5-a7ba-cf7e53f95e89","submission_id":"32ec6498-6ff8-4b4f-a934-c11907f955c1","submission_revision_id":"87710418-b7a3-4428-b733-743d5fa49f67","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:40:03.383820Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}