{"submission_id":"6882f51a-ddda-4122-803d-87367ba34b80","submission_revision_id":"10beb61e-f692-409b-8011-93e5988f6cf5","revision_version":1,"submission_status":"under_review","submitted_at":"2026-09-28T20:17:43.425060Z","submitter_name":"Jihye Roh","submitter_email":"jroh99@mit.edu","task":{"status":"submitted","version":1,"extra_fields":{},"submitted_at":"2026-09-28T20:17:43.425060+00:00","industry_domain":"Chemistry & Cheminformatics","task_short_name":"chem_patent_reaction_extraction_1","operating_system":"Linux","task_description":"Objective: Extract one reaction from a pharmaceutical patent into a structured JSON record, with full stereochemistry for every species.\n\nStarting State: No input files other than the task prompt (`input/task_prompt.md`). The prompt names a pharmaceutical patent and one paragraph of its experimental section; the agent has to find the patent on the web itself.\n\nInstructions & Rules: Read the named paragraph, identify the reaction it describes, and parse it into structured reaction data.\n- For each species, give its compound ID, role (reactant / product / agent), canonical SMILES, molecular formula, and amount in the mass or volume unit recorded in the patent.\n- For the reaction, give the time, temperature, yield and reaction SMILES.\n- Any molecule that contributes atoms to the product is a reactant; everything else is an agent.\n\nThe paragraph refers to its two reactants and the product only by compound number. The patent does draw every numbered compound, but as a structure image placed above the paragraph that prepares it, and the image filenames run on their own numbering, unrelated to the compound numbers. So the agent has to locate the preparation paragraph for each compound number, pick out the drawing that belongs to it, and read that drawing into a canonical SMILES string with every stereocenter right; one misread wedge inverts a center. The structures can also be reconstructed from the synthesis text instead, which for one reactant means following a chain of three earlier preparations back to the purchased material. A plain text fetch of the patent returns no structures at all.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Hard gates (score 0): the file is missing, is not `.json`, cannot be parsed, lacks the `compounds` or `reaction_metadata` keys, `compounds` is not a list of exactly 5 entries, any compound entry is missing one of `compound_ID`, `role`, `canonical_SMILES`, `molecular_formula` or `amount`, or `reaction_metadata` is missing one of `time`, `temperature`, `yield` or `reaction_smiles`. Extra keys beyond those required are ignored.\n\nField checks (24 fields):\n- Each reference compound is matched to a submitted entry by canonical SMILES, falling back to compound ID. For each of the 5 compounds, 4 fields are checked:\n  - `role`: exact.\n  - `canonical_SMILES`: exact string match to the RDKit 2026.3.6 canonical SMILES, stereochemistry included.\n  - `molecular_formula`: same element counts.\n  - `amount`: the same number, in the unit recorded in the patent (a missing or different unit scores 0).\n- `time`: 3 days and `temperature`: 80 °C, both in the recorded unit (no conversion, so hours or kelvin score 0).\n- `yield`: 62%, with the % sign.\n- `reaction_smiles`: each of the reactant, agent and product sides must contain the same canonical SMILES as the reference, in any order.\n\nAll four numeric comparisons are exact: the tolerance absorbs only float formatting, so `62` and `62.0` match but `62.3` does not.\n\nScore: a fully correct answer scores 1.0. Anything less is graded on a partial-credit scale worth 0.5 in total, i.e. 0.5 × (fields correct / 24).","software_and_version":"Web browser (Google Patents), Python 3.11, RDKit 2026.3.6","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"760044a4-a973-41ae-9dee-4e970153927b","filename":"task_prompt.md","category":"input_materials","content_type":"text/markdown","size_bytes":2366},{"file_id":"5e6219a2-08ff-4420-8e82-ada8ff7613ca","filename":"submission.json","category":"reference_output","content_type":"application/json","size_bytes":1443}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, coherent, objectively gradable, and reachable with the declared open-source toolchain.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly identifies the patent paragraph, extraction workflow, role definitions, output location, and exact JSON schema.\n- Inputs: Complete — The solver-visible prompt identifies the public patent and paragraph needed to obtain the source evidence.\n- Toolchain: Complete — Linux, browser or HTTP retrieval, Python 3.11, and the specified RDKit version provide a legally obtainable workflow.\n- Reference output: Complete — The JSON parses successfully, contains exactly five compounds with all required fields, and has complete reaction metadata. Its reaction SMILES exactly reproduces the listed species by role.\n- Rubric: Complete — Hard gates, 24 field checks, exact units, stereochemical string matching, partial-credit behavior, and the success threshold are explicit.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. Both `task_prompt.md` and `submission.json` were inspected in full; their formats and byte counts agree with the manifest, both are valid UTF-8, and the reference has the required schema. No attachment content was left unreviewed. The external patent page was not independently inspected within this evidence-only review, and independent RDKit recanonicalization was unavailable in the review environment; these are reviewer-tooling limitations rather than submission defects. The end-to-end path is clear: retrieve the identified public patent, inspect the named paragraph and relevant structure drawings or precursor preparations, canonicalize the five species with the specified RDKit version, write the required JSON, and evaluate it against the supplied reference and field-level rubric. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission provides a strong professional chemistry-extraction task, a structurally consistent reference, and precise evaluation criteria. No changes are required.","reviewed_at":"2026-09-29T15:17:45.106855Z"},"claim":null}