{"claim":{"id":"1d90163c-70ae-40d4-8259-29e4772bf710","submission_id":"0b089d3f-5317-41cc-bef9-7822c2b1ed2b","submission_revision_id":"010601f4-e6c2-44f8-b9e6-b20b4a3a19b1","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:45:17.986799Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"0b089d3f-5317-41cc-bef9-7822c2b1ed2b","submission_revision_id":"010601f4-e6c2-44f8-b9e6-b20b4a3a19b1","revision_version":1,"submission_status":"under_review","submitted_at":"2026-08-05T19:06:48.930902Z","submitter_name":"Tuan Vinh","submitter_email":"vinhtb@nih.gov","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"GPT 5.6 Sol","harness":"XHigh","estimated_score":"10–20%"}},"submitted_at":"2026-08-05T19:06:48.930902+00:00","industry_domain":"Computational Chemistry / Reaction Informatics","task_short_name":" reaction_plausibility_1000_instance_1","operating_system":"Windows, Linux","task_description":"Objective: Classify 1,000 proposed chemical reactions as plausible (1) or implausible (0) under reasonable conditions.\nStarting State: The input archive contains input/reaction_plausibility_input.csv with exactly two columns: case_id and reaction_smiles. Each reaction is formatted as reactants>>product. A blank submission template is also provided.\nInstructions & Rules: Evaluate bond changes, atom and valence consistency, mechanistic feasibility, functional-group compatibility, electronics, sterics, selectivity, stereochemistry, likely intermediates, and competing pathways. Do not reject a reaction solely because catalysts, additives, acids, bases, or solvents that contribute no heavy atoms are omitted. Preserve case_id,predicted_plausibility_label, include every case exactly once, and use only 0 or 1. Do not provide confidence scores, explanations, or extra columns.\nRequired Output: One completed CSV named predictions_template.csv containing exactly 1,000 prediction rows.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"case_id,predicted_plausibility_label, all 1,000 expected IDs exactly once, and only labels 0 or 1. Compare each prediction with the private expert label. Report accuracy, Matthews correlation coefficient, balanced accuracy, class recalls, and TP/TN/FP/FN. Missing, extra, or duplicate IDs, blank labels, or other values invalidate the submission.","software_and_version":"Python 3.12+; RDKit 2025.09","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"537be947-8d2e-48d3-8d51-fa1e1ca88a76","filename":"reaction_plausibility_input.csv","category":"input_materials","content_type":"text/csv","size_bytes":92298},{"file_id":"9c8e781d-016f-43c8-b240-229a3672ef95","filename":"reaction_plausibility_reference.csv","category":"reference_output","content_type":"text/csv","size_bytes":12021}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, coherent, objectively scorable, and reachable; the absent blank template is a nonmaterial packaging omission because its schema is fully specified.\n\n**Completeness Check:**\n- Instruction: Complete — The objective, classification criteria, omission policy, required filename, row count, schema, and allowed labels are explicit.\n- Inputs: Complete — The input CSV contains exactly 1,000 unique, sequential case IDs and nonblank reaction SMILES with valid reactant/product separators.\n- Toolchain: Complete — Python and RDKit are legally obtainable open-source tools compatible with the stated operating systems.\n- Reference output: Complete — The reference CSV contains every expected ID exactly once, with 500 labels in each class and no missing or invalid labels.\n- Rubric: Complete — Structural invalidation rules and accuracy, Matthews correlation coefficient, balanced accuracy, class recall, and confusion-matrix metrics provide objective evaluation.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both `reaction_plausibility_input.csv` and `reaction_plausibility_reference.csv` were inspected. They are readable UTF-8 CSV files matching their manifested sizes, with consistent two-column schemas, 1,000 data rows each, aligned IDs, no duplicate reactions, and structurally balanced reaction notation. Representative positive and negative reactions were reviewed and showed chemically meaningful distinctions. No attachment content remained unreviewed, although full RDKit semantic parsing was unavailable in the review environment; this is a reviewer-tooling limitation rather than a submission defect. A solver can classify each input reaction, create `predictions_template.csv` from the specified schema, and the evaluator can join it to the evaluator-only reference labels and compute all stated metrics. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports a screening label; public-benchmark provenance could not be independently searched under the permitted evidence boundary.\n\n**Recommended Changes:**\nThe submission provides a clean, balanced input/reference pair and a usable scoring contract. No material changes are required. Supplying the mentioned blank template would be optional packaging polish.","reviewed_at":"2026-08-14T20:45:37.762082Z"},"claim":{"id":"1d90163c-70ae-40d4-8259-29e4772bf710","submission_id":"0b089d3f-5317-41cc-bef9-7822c2b1ed2b","submission_revision_id":"010601f4-e6c2-44f8-b9e6-b20b4a3a19b1","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:45:17.986799Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}