{"submission_id":"5ae5b3e4-d6c1-4b91-91cd-c09e11a14123","submission_revision_id":"bd2f550a-4747-4b69-a290-f9ea90814bae","revision_version":2,"submission_status":"under_review","submitted_at":"2026-09-28T18:22:32.155214Z","submitter_name":"Jixian Liu","submitter_email":"jliu376@jh.edu","task":{"status":"submitted","version":2,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"5–10%"}},"submitted_at":"2026-09-28T18:22:32.155214+00:00","industry_domain":"Robotics Engineering","task_short_name":"multi_robot_convoy_navigation","operating_system":"Linux","task_description":"Objective: Port a lab's ROS 1 four-robot demo to ROS 2 and make it carry out warehouse missions issued at runtime by a fleet manager, inside a provided 2-D multi-robot simulator.\n\nStarting state: An Ubuntu 22.04 VM with ROS 2 Humble and Navigation2 installed, no internet access, sudo available. The output/source directory already contains a ROS 2 project: a Stage-like simulator + fleet manager + scorer (fleetbench/), a 16 m x 10 m warehouse site (sites/warehouse/), and a ROS 2 port of the lab's original ROS 1 Melodic demo for four TurtleBot3 Burgers (src/mbot, src/mbot_follower). The original demo used one AMCL + move_base\nper robot on a shared map, and TF-driven leader-follower formation with a near-identity feedback-linearisation controller. The port keeps these methods and their original weaknesses (delivered \"as is\" so the agent has something to diagnose, not something already correct). Two example missions (input/missions/dev_1.json, dev_2.json), the machine-readable rules\n(input/task_contract.json), the original ROS 1 source and manual (input/legacy_ros1_example/), and the full brief (input/TASK.md) are also provided read-only.\n\nInstructions & rules: Every mission has three phases sent one at a time on the ROS 2 topic /fleet/mission: (1) assemble - localise each robot from its dock/station and bring it to its slot in the first convoy formation; (2) convoy - the leader drives a given polyline route while the other three robots keep a rigid offset formation in the open aisles and switch to a single-file column (in a given order and spacing) to pass through an 0.8 m fire door, switching back afterwards; (3) disperse - each robot goes to its own delivery station or dock. Only the two packages src/mbot and src/mbot_follower may be changed (parameters, launch files, new nodes);\nevery other file of the project must stay byte-identical, and the packages may only contain small text/config files (no binaries, no prior maps). The agent must keep the lab's two methods (per-robot AMCL+Nav2 localisation/navigation, and feedback-linearisation formation control) but may retune, extend or add nodes to them. Build the two packages offline outside the output directory (colcon), run both development missions in full with `python3 -m fleetbench.cli run`,\nand deliver: the modified source, the two development runs, a metrics.json summarising each run's checks, a report.md (Diagnosis / Changes / Results, >=250 words), and a submission.json index. No hard-coded knowledge of any specific mission's routes, stations or obstacles - the stack may use only the site map file and the live phase messages.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The evaluator runs on the VM as root, after the agent finishes, and every check below is computed automatically by the grader from files/processes it produces or observes itself - never from anything the agent reports about its own success.\n\nSetup: it stages 4 HIDDEN missions plus its own sealed copies of the site and development missions into a root-only directory, verifies with dpkg that no ROS/Python system package was modified, then rebuilds ONLY src/mbot and src/mbot_follower offline as a throwaway unprivileged user.\n\nRe-execution, per hidden mission, in one private ROS domain over loopback UDP:\n  - root runs the public fleetbench trial verbatim (simulator + fleet manager + ground-truth recorder);\n  - the unprivileged user runs `ros2 launch mbot fleet.launch.py map:=<map.yaml>`. The map:= argument is an ordinary, world-readable copy of the site map (so nav2_map_server/AMCL work exactly as on any normal ROS 2 stack) - this is explicitly NOT the same thing as the evaluator's sealed reference copy of the site, which is used only to build hidden ground truth and which the\n    unprivileged process cannot read, along with the mission file and any other evaluator output. The stack learns each phase only from the live /fleet/mission topic when it starts.\n\nScore = 0.05*S1 + 0.05*S2 + 0.90*S3, all automatic:\nS1 (contract): submission.json well-formed; output tree has no symlinks/special files; every file outside the two editable packages is byte-identical (SHA-256) to the frozen project; the two packages contain only small UTF-8 text/config files; the sandboxed offline build succeeds.\n\nS2 (development evidence, all re-derived by the grader): each of the 2 development runs must be of the published mission, a FULL run at real-time factor 1 by the unmodified public runner, with recorded source_hashes.json equal to the SHA-256 of the submitted packages; the grader re-scores the\nrecorded ground-truth trial from scratch and checks it matches result.json exactly; metrics.json must reproduce those numbers; report.md must contain the 3 required sections.\n\nS3 (the task itself, 4 hidden missions x 9 automatic checks, all from the grader's own 20 Hz ground-truth pose/velocity log): final pose within 0.10 m/0.20 rad of target and stopped; leader reaches the route end within 0.15 m cross-track at >= the mission's minimum speed; each follower's RMS/max distance to its offset slot (<=0.10 m RMS/0.25 m max) or, in column formation, to the leader's actual driven path (<=0.06 m RMS cross-track, <=0.12 m RMS along-path spacing); zero collisions and >=0.30 m minimum separation between any two robots throughout. S3 = 0.4*(fraction of the 36 checks passed) + 0.6*[all 36 passed], so partial/inconsistent solutions score far below a solution that reliably meets every requirement on unseen missions.","software_and_version":"ROS 2 Humble (Navigation2 1.1.20: nav2_amcl, NavFn, DWB, BT navigator), Python 3.10","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"b2bb4f71-ac84-4236-b8da-75f737834362","filename":"inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":1677225},{"file_id":"7bc4d349-0202-4860-a96c-3cd75fbfad41","filename":"reference.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":14888}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, safely reachable, professionally challenging, and supported by a deterministic end-to-end evaluation path.\n\n**Completeness Check:**\n- Instruction: Complete — `TASK.md`, `task_contract.json`, and interface documentation clearly define the three phases, editable scope, runtime constraints, required methods, deliverables, and prohibited shortcuts.\n- Inputs: Complete — The ROS 2 project, two development missions, warehouse map and geometry, legacy ROS 1 source, manual, screenshots, and output examples are present and readable.\n- Toolchain: Complete — Ubuntu 22.04, ROS 2 Humble, Navigation2 1.1.20, Python 3.10, offline build requirements, launch command, CPU expectations, and runtime procedure are specified.\n- Reference output: Complete — The protected reference provides matching development missions and site data, four held-out missions, a frozen-project hash manifest, and the evaluation protocol. Behavioral scoring appropriately replaces a conventional golden output.\n- Rubric: Complete — Ground-truth checks, tolerances, weights, phase deadlines, safety rules, partial-credit formula, source-integrity checks, and evidence-validation requirements are machine-readable and objectively defined.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`. I inspected `inputs.zip` and `reference.zip`, including their complete inventories, integrity, metadata, encodings, schemas, and representative content. The archives contain 70 and 11 entries respectively, are unencrypted, have no duplicate or traversal paths, and extract to ordinary files without links or special entries. Relevant inspected members included `TASK.md`, both task-contract copies, both development missions, output examples, project documentation, simulator/runner/scorer source, both editable starter packages, site JSON/YAML/PGM data, the legacy source and `手册.docx`, all six legacy screenshots, `protocol.json`, `project_manifest.json`, matching protected development/site copies, and all four held-out missions. JSON documents parsed successfully; text is ASCII or UTF-8; image and document formats match their extensions. The protected development missions and site files are byte-identical to their public copies, and every frozen project hash matches `project_manifest.json`. Mission geometry checks found consistent route lengths, schedules, targets, time budgets, obstacle clearance, and feasible offset trajectories. No relevant content remained unreviewed; ancillary RViz and legacy launch configuration were inventoried and type-checked rather than analyzed line by line.\n\nThe completion path is coherent: modify only the two permitted packages, build them offline outside the output tree, launch against the supplied map and live mission topic, record both full development trials, produce the indexed evidence, and undergo an isolated unprivileged rebuild plus four sealed missions scored from 20 Hz ground truth. Uploaded programs were not executed under the review policy, and the review environment lacked the declared ROS/Python runtime; runtime behavior was therefore assessed statically. This is a reviewer-tooling limitation, not an attachment or task barrier. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone — no affirmative evidence supports an AI-synthetic, public-benchmark, or synthetic-defect-fixture label. Public identifiability was not independently checked.\n\n**Recommended Changes:**\nThe submission already provides unusually strong contracts, protected-evidence separation, mission diversity, integrity checks, and objective scoring. No changes are required.","reviewed_at":"2026-09-28T18:34:12.720561Z"},"claim":null}