{"claim":{"id":"46077fe6-fe05-45c7-a4f6-6a0125961f8c","submission_id":"000f9a8b-ab30-4423-b688-a02d25c8458e","submission_revision_id":"2a1e49a8-18c0-4315-aa8a-f522944854df","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:38:24.277448Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"000f9a8b-ab30-4423-b688-a02d25c8458e","submission_revision_id":"2a1e49a8-18c0-4315-aa8a-f522944854df","revision_version":4,"submission_status":"under_review","submitted_at":"2026-10-06T18:21:48.641160Z","submitter_name":"Lukas Kim","submitter_email":"kim_lukas@berkeley.edu","task":{"status":"submitted","version":4,"extra_fields":{},"submitted_at":"2026-10-06T18:21:48.641160+00:00","industry_domain":"Computational Chemistry","task_short_name":"bond_order_assignment_1","operating_system":"Windows, Linux, MacOS","task_description":"**Objective:** Assign bonds, bond orders and formal charges to drug-like molecules from their 3D coordinates alone, and write the result as a V2000 SDF that beats an RDKit `DetermineBonds` (xyz2mol) baseline.\n\n**Starting State:**\n- `input/structures.xyz`: a multi-frame extended XYZ file of 81,720 conformers with all hydrogens explicit. Realistically distorted geometries are included deliberately to verify the robustness of the bond assignment. Each frame has an opaque name (`mol_00000`, `mol_00001`, ...) and its total formal charge (`charge=` in the frame header).\n- `input/assign.py`: a script that reads the XYZ with ASE, calls `assign_connectivity(atoms)` on each frame and writes one SDF record per frame. `assign_connectivity` is a placeholder that returns no bonds.\n- A Python virtual environment at `env/` with ASE, RDKit, NumPy and SciPy (`requirements.txt`).\n\n**Instructions & Rules:**\n1. Work from the project root and use the bundled interpreter, `env/bin/python`.\n2. In `input/assign.py`, replace the body of `assign_connectivity(atoms)`. Each call receives one frame as an ASE `Atoms` object and must return:\n   - `bonds`: a list of `(i, j, order)` tuples, where `i` and `j` are 0-based atom indices and `order` is a V2000 bond type (1 single, 2 double, 3 triple).\n   - `charges`: a list of per-atom formal charges, in atom order.\n\n   The frame's total formal charge is in `atoms.info[\"charge\"]`. Do not change the function signature, `write_sdf_record` or `main`.\n3. Generate the output:\n   `env/bin/python input/assign.py -i input/structures.xyz -o output/generated.sdf`\n\nConstraints:\n- Handle each frame independently. A frame's result may depend only on that frame's element symbols, coordinates and total charge. Do not compare, cluster or match frames against each other, do not reuse one frame's assignment for another, and do not cache results across frames.\n- Do not use any reference connectivity (no reference SDF, scoring script or source dataset).\n- Do not modify the input XYZ file.\n\n**Scoring:** A grader pairs output and reference records by position. For each record it takes every atom pair bonded on either side (unordered; a pair missing on one side counts as type 0) and counts a bond as matched only if its type equals the reference type exactly. The reference is Kekulé (types 1, 2, 3 only), so aromatic type 4 never matches. Equivalent Kekulé structures are not penalized: the grader may flip any ring whose bonds alternate single/double (rings of any size or element, repeatedly, so fused systems are covered) and scores the flipped assignment that best matches the reference. Stereo flags and formal charges are not scored. Two metrics are pooled over all 81,720 records: **bond %** (matched bonds / all bonded pairs) and **mol %** (records with every bond matched / all records). The baseline scores 89.39% bond and 59.48% mol. To pass, both metrics must be strictly higher.\n\n**Deliverable:** `output/generated.sdf` (plus the modified `input/assign.py`).","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Score the agent output with:\n```sh\n    env/bin/python reference/compare.py -o output/generated.sdf -r reference/reference.sdf\n```\nAll three rules must pass. They are checked in order, and the first failure ends the evaluation.\n\n1. **File exists:** `output/generated.sdf` is present.\n2. **Records align with the input:** `compare.py` finishes and prints a score without raising an error. This requires exactly 81,720 records, in input order, each with the same name (line 1) and atom count as the matching reference record. Any V2000 record over 999 atoms or bonds is an error.\n3. **Beats the baseline:** On the `total` row, **bond % > 89.39** and **mol % > 59.48** (both strictly greater).\n   - bond %: matched bonds / union of bonded atom pairs, pooled over all records. A pair bonded on only one side counts as type 0, so missing and extra bonds are both errors. Types are compared literally (aromatic 4 never matches a Kekulé 1 or 2), except that any ring alternating single/double in the output, of any size or element, may be flipped to the equivalent Kekulé structure that best matches the reference.\n   - mol %: records in which every bond matches / all records.\n   - Stereo flags and formal charges are not scored.\n\nBaseline (RDKit `rdDetermineBonds.DetermineBonds` with Kekulization, falling back to a 1.2 × covalent-radii distance cutoff, see `reference/baseline_solution.py` and `reference/baseline_result.txt`):\n\n| records | bond % | mol % |\n|---|---|---|\n| 81,720 | 89.39 | 59.48 |","software_and_version":"Python 3.14.6 with ASE 3.29.0, RDKit 2026.3.6, NumPy 2.5.3 and SciPy 1.18.1","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"81935337-b966-4ac7-9821-3de6dbc65608","filename":"assign.py","category":"input_materials","content_type":"text/x-python-script","size_bytes":2173},{"file_id":"2bb4a8db-9207-4c7a-bf16-61f0cee85b9a","filename":"requirements.txt","category":"input_materials","content_type":"text/plain","size_bytes":257},{"file_id":"4866bf4c-b941-4a87-981f-8dc8a8e5a8ba","filename":"structures.xyz","category":"input_materials","content_type":"chemical/x-xyz","size_bytes":158188671},{"file_id":"f0213a77-43d0-46f1-8efc-7d2e08b29d12","filename":"baseline_result.txt","category":"reference_output","content_type":"text/plain","size_bytes":216},{"file_id":"112fd3ea-928c-40f5-8dde-db6b9acf5a2c","filename":"compare.py","category":"reference_output","content_type":"text/x-python-script","size_bytes":7996},{"file_id":"0788ef73-1c3e-4777-95a5-751ae9b65d6d","filename":"baseline_solution.py","category":"reference_output","content_type":"text/x-python-script","size_bytes":3728},{"file_id":"298fba40-fc03-43db-a08c-7379592d5a63","filename":"baseline.sdf","category":"reference_output","content_type":"chemical/x-molfile","size_bytes":309856844},{"file_id":"88163e96-5d1c-4aea-83b3-c74b0b5d54f3","filename":"reference.sdf","category":"reference_output","content_type":"chemical/x-molfile","size_bytes":321351462}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — the revision is complete, internally consistent, and provides a reachable workflow with objective evaluation.\n\n**Completeness Check:**\n- Instruction: Complete — defines the per-frame function contract, constraints, command, and deliverable.\n- Inputs: Complete — the full XYZ is readable and contains 81,720 named frames; maximum frame size is 159 atoms.\n- Toolchain: Complete — pinned ASE, RDKit, NumPy, and SciPy dependencies are specified and open source.\n- Reference output: Complete — baseline output, reference output, and comparator are supplied in the reference-output category.\n- Rubric: Complete — record alignment and exact bond-type scoring are specified, with strict thresholds against the reported baseline.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. I inspected all eight attachments: `assign.py`, `requirements.txt`, `structures.xyz`, `baseline_result.txt`, `compare.py`, `baseline_solution.py`, `baseline.sdf`, and `reference.sdf`. The downloaded byte counts match the manifest. Full structural scans found 81,720 frames and records, no name or atom-count mismatches, and V2000 records below the format limit; reference bond types are 1, 2, or 3. The reference files are categorized separately from solver inputs. No content was left unreviewed; submitted scripts were inspected as text and not executed. No reviewer-tooling limitations remain. The path from per-frame XYZ input through the required SDF output to objective comparison is clear. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. Public-benchmark status was not independently verified.\n\n**Recommended Changes:**\nThe revision provides a clear, technically specific task and usable evaluation materials. No changes are required.","reviewed_at":"2026-10-06T18:30:01.159830Z"},"claim":{"id":"46077fe6-fe05-45c7-a4f6-6a0125961f8c","submission_id":"000f9a8b-ab30-4423-b688-a02d25c8458e","submission_revision_id":"2a1e49a8-18c0-4315-aa8a-f522944854df","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:38:24.277448Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}