{"claim":{"id":"71676634-353e-4683-9b4d-5482e6a798c2","submission_id":"0e8f173b-c4a9-4d0b-b6f1-8d775c348b54","submission_revision_id":"5c818f81-5b18-465f-a7cf-6f3423316f84","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:45:27.738790Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"0e8f173b-c4a9-4d0b-b6f1-8d775c348b54","submission_revision_id":"5c818f81-5b18-465f-a7cf-6f3423316f84","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-06T06:22:26.994780Z","submitter_name":"Tuan Vinh","submitter_email":"vinhtb@nih.gov","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"model":"GPT 5.6 Sol XHigh","harness":"Codex","estimated_score":"10–20%"}},"submitted_at":"2026-08-06T06:22:26.994780+00:00","industry_domain":"Computational Chemistry / Reaction Informatics","task_short_name":"exact_product_prediction_100_instance_1","operating_system":"Windows, Linux","task_description":"Objective: Predict the exact product for each of 100 chemical reactions from reactant-only SMILES.\nStarting State: The input is one Excel workbook named `product_prediction_100_input.xlsx`. It contains exactly one worksheet with 100 data rows and exactly two columns in this order: `case_id`, `reactant_smiles`. Reactant SMILES may contain multiple dot-separated components.\nInstructions and Rules: Predict one product SMILES for every case and create one Excel workbook containing exactly two columns in this order: `case_id`, `predicted_product_smiles`. Preserve every case ID exactly and include each case ID once. Predictions must be nonempty, RDKit-parseable SMILES and must preserve product connectivity, formal charge, bond order, and defined stereochemistry. Do not add, remove, or reorder columns. Do not include explanations, energies, transition states, geometries, or laboratory instructions.\nRequired Output: Submit one Excel workbook named `product_prediction_100_output.xlsx`. It must contain exactly one worksheet, the required header row, and exactly 100 prediction rows.\nExpected Completion Time: up to 3 hours for an expert with appropriate cheminformatics tools.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The submission must be an Excel workbook named `product_prediction_100_output.xlsx` containing exactly one worksheet. The worksheet must contain exactly two columns in this exact order: `case_id`, `predicted_product_smiles`, followed by exactly 100 data rows. The `case_id` values must be exactly `case_0001` through `case_0100`, with every expected ID present once and no missing, extra, blank, or duplicate IDs. Any additional worksheet, additional column, missing ID, extra ID, blank ID, or duplicate ID invalidates the submission.\nMatch submitted rows to the reference by `case_id`; row order does not affect scoring. For each submitted and reference product, parse `predicted_product_smiles` using RDKit with sanitization enabled. Blank or invalid SMILES are incorrect. For each successfully parsed molecule, clear all atom-map numbers, remove only RDKit-removable explicit hydrogens using `Chem.RemoveHs`, and generate canonical isomeric SMILES with `Chem.MolToSmiles(mol, canonical=True, isomericSmiles=True)`. Preserve formal charge, bond order, isotopes, defined stereochemistry, and every disconnected product component; do not strip salts or retain only the largest fragment.\nA case is correct only when the submitted and reference canonical isomeric SMILES are identical. Report exact-product accuracy as the number of correct cases divided by 100. Also report the integer count of correct cases, invalid or blank predictions, missing IDs, extra IDs, and duplicate IDs. Full success is 100/100 correct with no structural workbook errors.","software_and_version":"Python 3.12+; RDKit 2024.03 or later","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"32f1ba17-3d2a-4ccf-b9de-d87263fee9dd","filename":"product_prediction_100_input.xlsx","category":"input_materials","content_type":"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet","size_bytes":5994},{"file_id":"868396a5-72cb-4712-9327-00409bbff0ce","filename":"product_prediction_100_reference_output.xlsx","category":"reference_output","content_type":"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet","size_bytes":6018}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, internally consistent, safely reachable, and objectively evaluable, with only a reviewer-tooling limitation on independent RDKit sanitization.\n\n**Completeness Check:**\n- Instruction: Complete — The required prediction task, filenames, workbook schema, row count, chemical fidelity requirements, and prohibited additions are explicit.\n- Inputs: Complete — `product_prediction_100_input.xlsx` is a readable, unencrypted OOXML workbook with one worksheet, the required two columns, 100 nonblank rows, and every expected case ID exactly once.\n- Toolchain: Complete — Python and RDKit are free, legally obtainable, cross-platform tools; routine Excel-writing support can be added without contributor-owned information.\n- Reference output: Complete — `product_prediction_100_reference_output.xlsx` has one worksheet, the required schema, 100 nonblank reference products, and the identical complete case-ID set.\n- Rubric: Complete — Workbook validity, ID integrity, RDKit parsing and normalization, exact structural comparison, partial accuracy, and failure counts are defined objectively.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both `product_prediction_100_input.xlsx` and `product_prediction_100_reference_output.xlsx` were inspected. Their declared and downloaded sizes agree; each is a valid eight-member OOXML container with no encryption, macros, formulas, external relationships, embedded executables, unsafe archive paths, or unexpected worksheets. All rows and cells were structurally inspected, representative chemistry was reviewed, and lexical checks found balanced SMILES branches, brackets, and ring closures. No attachment content remained unreviewed. A local RDKit runtime was unavailable, so sanitization and canonicalization could not be independently executed; this is a reviewer-tooling limitation, not evidence of a submission defect. The completion path is clear: read the reactants, predict one product per ID, write the specified workbook, then validate IDs and compare RDKit-canonicalized isomeric SMILES against the evaluator-only reference. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. The chemistry is unusual, but the supplied evidence does not affirm AI fabrication, public-benchmark identity, or an injected-defect task. Public provenance was not independently searched because external lookup is outside this review.\n\n**Recommended Changes:**\nThe submission provides a coherent input, reference, output contract, and exact scoring procedure. No changes are required.","reviewed_at":"2026-08-14T20:46:02.986524Z"},"claim":{"id":"71676634-353e-4683-9b4d-5482e6a798c2","submission_id":"0e8f173b-c4a9-4d0b-b6f1-8d775c348b54","submission_revision_id":"5c818f81-5b18-465f-a7cf-6f3423316f84","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:45:27.738790Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}