{"claim":{"id":"d1597e94-3af5-4cdd-b6c2-d0ee172134a5","submission_id":"688f1beb-e9e9-413b-98d1-d1575e47ee03","submission_revision_id":"fc7e84d0-769b-4bdf-a93c-b0aeb71ca0ad","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:04:12.938399Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"688f1beb-e9e9-413b-98d1-d1575e47ee03","submission_revision_id":"fc7e84d0-769b-4bdf-a93c-b0aeb71ca0ad","revision_version":4,"submission_status":"under_review","submitted_at":"2026-08-16T03:17:59.224113Z","submitter_name":"Yongshuo Zong","submitter_email":"yongshuo.zong@gmail.com","task":{"status":"submitted","version":4,"extra_fields":{"difficulty_self_test":{"model":"Gemini Flash 3.6","estimated_score":"5–10%"}},"submitted_at":"2026-08-16T03:17:59.224113+00:00","industry_domain":"Genomics & Sequence Analysis","task_short_name":"spatial_transcriptomics_instance_1","operating_system":"Linux","task_description":"Objective: Perform spatial transcriptomics analysis on a 10x Genomics Visium dataset from human brain tissue to identify spatially coherent tissue domains corresponding to cortical layers.\n\nStarting State: A working directory containing raw 10x Space Ranger output files (no preprocessing):\n\ninput/filtered_feature_bc_matrix.h5 — Filtered count matrix (~3,600 spots × ~33,500 genes, 12 MB)\ninput/spatial/ — Spatial metadata: spot coordinates CSV (no header), H&E tissue images, scale factors JSON\noutput/ — Empty output directory\ntask_prompt.txt — Instructions\nsoftware/main.py — Evaluation script\n\nPython 3 is available. The agent may install packages using pip (recommended: scanpy>=1.10.0, anndata>=0.10.0, squidpy>=1.4.0).\n\nInstructions & Rules:\n\n- Load the Visium data from the raw h5 file and attach spatial coordinates from the tissue positions CSV (which has no header row)\n- Perform quality control — the agent must decide appropriate filtering thresholds based on data distributions (no config file is provided)\n- Normalize, select variable genes, run PCA, build neighbor graph, compute UMAP, and cluster with Leiden\n- Choose a clustering resolution appropriate for cortical layer detection (the tissue has a layered architecture with ~7 distinct domains)\n- Find differentially expressed marker genes per cluster\n- Write all required outputs to output/: QC summary (qc_stats.txt), cluster assignments (cell_clusters.csv), marker genes (marker_genes.csv), UMAP plot (umap_plot.png), and processed AnnData (results.h5ad)\n\n\nData Source & License\nDataset: Human dorsolateral prefrontal cortex (DLPFC), sample 151673\nSource: Maynard KR et al. \"Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex.\" Nature Neuroscience 24, 425–436 (2021). PMID: 33558015\nPlatform: 10x Genomics Visium Spatial Gene Expression\nData Host: Lieber Institute for Brain Development (spatial-dlpfc.s3.us-east-2.amazonaws.com)\nLicense: Creative Commons Attribution 4.0 (CC BY 4.0)\nGround Truth: Pathologist-annotated cortical layer labels from the original study (7 classes: L1, L2, L3, L4, L5, L6, WM)\n\nSolver isolation: The solver must run fully offline after initial dependency installation. Internet access should be disabled or strictly allowlisted to package repositories only during the solver phase to prevent retrieval of the public annotation labels.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Compare the agent's outputs against reference using the evaluation script software/main.py. Scoring is a weighted combination of six components, all fully deterministic.\n\nQC Stats (15% weight): The file output/qc_stats.txt must parse as key-value text (using : or = delimiters). It must report a retained spot count (via a key like spots_after) in the range 2000–3700. Full credit if parseable and in range; partial credit (0.3) if parseable but the count is missing or outside range.\n\nCell Clusters (15% weight): The file output/cell_clusters.csv must contain barcode and cluster columns. Predicted barcodes are validated against the input matrix — barcodes not found in the input incur a penalty. The file must contain 2000–3700 unique spots with 4–15 clusters. Duplicate barcodes are detected and penalized (0.7× multiplier).\n\nMarker Genes (10% weight): The file output/marker_genes.csv must contain cluster and gene columns. The score (or logfoldchange) and pval_adj columns are checked when present: score values must be numeric, and p-values must be numeric in [0, 1]. Each cluster should have at least 5 marker genes. Coverage score = fraction of clusters with ≥5 markers, multiplied by a column-quality factor.\n\nUMAP Plot (5% weight): The file output/umap_plot.png must be a valid PNG larger than 2000 bytes. Full image decode is performed when PIL is available; otherwise, PNG header validation is used (0.8× credit).\n\nResults H5AD (15% weight): The file output/results.h5ad must be a valid AnnData HDF5 file containing X_pca in .obsm (0.33), X_umap in .obsm (0.33), and a clustering column (leiden, cluster, or louvain) in .obs (0.34).\n\nClustering ARI (40% weight): Adjusted Rand Index between the agent's predicted cluster assignments and 3,611 pathologist-annotated cortical layer labels (L1–L6 + WM). Predicted barcodes must overlap at least 50% of reference barcodes; below this threshold, the ARI score is zero. A coverage penalty is applied when overlap falls below 90% of reference barcodes (score × coverage/0.90). Full credit at ARI ≥ 0.50. Linear partial credit between ARI 0.25 and 0.50, calculated as (ARI − 0.25) / 0.25. Zero credit below ARI 0.25.\n\nFinal Score = Σ (weight × component_score). Maximum = 1.0.\n","software_and_version":"Python, scanpy","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"6b67cdcd-203e-45bf-96f6-8c9522aa5fc4","filename":"reference_labels.csv","category":"reference_output","content_type":"text/csv","size_bytes":79456},{"file_id":"ea2ccce9-a6e5-4f2b-9891-130e67b47f66","filename":"task_prompt.txt","category":"input_materials","content_type":"text/plain","size_bytes":3950},{"file_id":"c11a5f9f-2a63-4dbc-915c-8fdbd5066f69","filename":"input.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":17889479},{"file_id":"946ffe72-fecb-489b-904d-cf8e96ee35e0","filename":"main.py","category":"reference_output","content_type":"text/x-python-script","size_bytes":17063}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, professionally scoped, objectively evaluable, and executable on a standard single-machine Python environment, with only non-material reviewer-tooling uncertainty around deep HDF5 parsing.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly specifies preprocessing, QC, normalization, dimensionality reduction, clustering, marker analysis, visualization, and five required output artifacts.\n- Inputs: Complete — The archive is intact and contains the count matrix, 4,992 spatial-position records including 3,639 in-tissue spots, valid scale factors, and two valid tissue images.\n- Toolchain: Complete — Python and the named open-source genomics packages provide a legally obtainable, single-machine workflow suitable for this dataset’s scale.\n- Reference output: Complete — The reference contains 3,611 unique barcodes spanning exactly seven labels (L1–L6 and WM); all reference barcodes occur among the in-tissue positions.\n- Rubric: Complete — The evaluator implements the stated six components, weights summing to 1.0, artifact validation, barcode checks, coverage penalties, and ARI thresholds.\n\n**File and Task Reachability:**\nAttachment classification: `partially_reachable`. Task classification: `reachable`. The inspected attachments were `reference_labels.csv`, `task_prompt.txt`, `input.zip`, and `main.py`. Every archive member was also inspected: `filtered_feature_bc_matrix.h5`, `scalefactors_json.json`, both tissue PNGs, and `tissue_positions_list.txt`. Manifest byte counts matched, the archive passed integrity checks, the images fully decoded and visually showed the expected tissue section, the coordinate and reference schemas were validated, and the HDF5 file had a valid signature and expected 10x schema identifiers. The reviewer environment lacked an HDF5 parser, so the matrix’s internal dimensions and numerical arrays could not be independently decoded; this is a reviewer-tooling limitation rather than evidence of a submission defect. No other content was unreviewed. The `.txt` position filename differs from the prompt’s `.csv` wording, but its contents use the required CSV schema and the evaluator explicitly supports both names. The end-to-end path is coherent: load the supplied matrix and coordinates, process and cluster with the declared Python stack, generate the five outputs, validate their schemas, and compare cluster assignments against evaluator-only labels. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The named DLPFC sample, publication, dataset, and associated layer annotations are publicly identifiable. The submission mitigates direct answer retrieval by keeping labels evaluator-only and requiring the solver to remain offline after dependency installation. There is no affirmative evidence that exact barcode-level labels are pervasively memorized.\n\n**Recommended Changes:**\nThe submission provides coherent authentic inputs, protected references, and comprehensive deterministic evaluation coverage. No material changes are required.","reviewed_at":"2026-08-17T04:33:39.958948Z"},"claim":{"id":"d1597e94-3af5-4cdd-b6c2-d0ee172134a5","submission_id":"688f1beb-e9e9-413b-98d1-d1575e47ee03","submission_revision_id":"fc7e84d0-769b-4bdf-a93c-b0aeb71ca0ad","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:04:12.938399Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}