{"submission_id":"0a421fe3-4818-408f-b2d0-b41c6d6ab2f6","submission_revision_id":"1173ca2c-11f6-4e88-a8a1-1af7186bd4c4","revision_version":2,"submission_status":"under_review","submitted_at":"2026-08-16T07:37:51.616035Z","submitter_name":"Ziao Yang","submitter_email":"ziaoyang@brandeis.edu","task":{"status":"submitted","version":2,"extra_fields":{},"submitted_at":"2026-08-16T07:37:51.616035+00:00","industry_domain":"Earth & Atmospheric Science / Scientific Machine Learning","task_short_name":"nexrad_nowcast_release_qualification_1","operating_system":"Linux","task_description":"You are the scientific ML engineer performing release qualification for a causal, short-range radar-reflectivity nowcast. The staged input is a derived, provenance-pinned subset of public NOAA NEXRAD Level-II KTLX observations from the 2024-05-06 severe-convection event. The data are modified Cartesian reflectivity grids, not unaltered NOAA products.\n\nRead `base/input/README_DATA.md` and `base/input/OUTPUT_CONTRACT.json`, then inspect `base/input/nexrad_public.npz`. It contains two complete training windows, one complete validation window, and only the six context scans for a test window. Each window requires three future-scan forecasts on a 96 x 96 dBZ grid. The test future is hidden. A runnable causal persistence baseline is provided at `base/software/nowcast_starter.py`.\n\nCopy or replace the starter as `base/output/pipeline.py` and build a deterministic nowcasting pipeline. Fit or select every parameter using only the complete training and validation windows. Your method may use motion estimation, advection, intensity evolution, blending, filtering, or another scientifically defensible NumPy implementation, but test predictions must use only `test_context_dbz` and its timestamps. Do not infer from filenames or provenance which hidden arrays were selected, and do not download data, code, models, or packages.\n\nRun your pipeline and leave exactly these release artifacts under `base/output/`:\n- `pipeline.py`: self-contained Python/NumPy implementation with a CLI documented by the output contract\n- `validation_predictions.npy`: float32 `[1,3,96,96]` forecasts produced from the visible validation context\n- `test_predictions.npy`: float32 `[1,3,96,96]` forecasts produced only from the test context\n- `verification.json`: visible-validation weighted MAE, CSI, and FSS values in the required schema\n- `model_manifest.json`: deterministic seed, algorithm, selected parameters, split usage, input digest, and environment\n- `release_report.md`: concise release review with data provenance, method and parameter-selection rationale, causality controls, per-lead validation findings, operational limitations, and a `release_decision: GO` or `release_decision: NO-GO` line\n\nThe evaluator snapshots the completed artifacts, loads protected truth into evaluator memory, deletes and verifies the reference tree, then replays your snapped pipeline unprivileged, resource-bounded, without network, and with only the public archive. All six replay artifacts must reproduce exactly; hidden truth is never present during replay. It scores test predictions against withheld future NEXRAD grids relative to persistence, recomputes every visible-validation metric, and cross-checks the manifest and report. Treat missing/no-echo pixels as -10 dBZ, keep predictions finite and within [-10,70] dBZ, preserve chronological order, and use no future target during inference.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The evaluator protects the newly staged reference tree and snapshots the candidate directory through a pinned directory descriptor using no-follow, nonblocking openat reads, regular-file/link-count/size checks, and before/after stability checks. It loads protected truth into evaluator memory, deletes and verifies the entire reference tree, then replays the snapped pipeline under CPU, memory, process, descriptor, and file-size limits as an unprivileged user in a no-network namespace containing only the public NPZ. Every replayed artifact must match the submitted artifact bytes exactly. It then validates file contracts and recomputes all visible-validation claims. Hidden quality is scored against the withheld test future relative to persistence. Score weights are: required safe artifacts and exact replay 15%; reproducible manifest and input binding 10%; exact visible-validation verification and report cross-check 15%; hidden weighted-MAE skill 25%; hidden 20-dBZ and 35-dBZ CSI skill 15%; hidden 5x5 fractions-skill performance 15%; scientific release report and causality/provenance controls 5%. Missing, unsafe, non-reproducible, non-finite, out-of-range, or wrong-shape predictions fail the corresponding gate. Passing requires score >= 0.75 plus exact replay, valid test predictions, and exact visible claims. Missing/corrupt protected references, failure to remove them, or failure to enforce replay isolation is an infrastructure error.","software_and_version":"Python 3.12.13, NumPy 2.3.5","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"c77bad13-26e2-4e33-9636-9461f0871f41","filename":"input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":197305},{"file_id":"11c52325-5678-4117-ac41-8a9f02076ee8","filename":"reference_output_and_evaluation_dependencies.zip","category":"reference_output","content_type":"application/zip","size_bytes":124167}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally representative, securely evaluable, and reachable end to end with no material defects.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly defines the scientific objective, causal split policy, six required artifacts, schemas, metric equations, release gates, and deterministic replay requirements.\n- Inputs: Complete — The public archive contains the stated training, validation, and test-context arrays with finite, bounded values, correct dimensions, chronological timestamps, coordinates, thresholds, provenance, and no test future.\n- Toolchain: Complete — CPython 3.12.13 and NumPy 2.3.5 are precisely pinned in a frozen, open-source, CPU-only environment suitable for one-machine execution.\n- Reference output: Complete — Protected test truth, an evaluation contract, full provenance, and six coherent expert artifacts are supplied separately. Independent metric recomputation exactly matched the visible claims, confirmed all release gates, and showed the reference forecast outperforming persistence on the protected metrics.\n- Rubric: Complete — The evaluator implements the stated weighted scoring, passing threshold, mandatory gates, artifact safety checks, exact replay, visible-claim verification, hidden-quality comparisons, and infrastructure-error handling.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both declared archives were downloaded at their manifest sizes, passed integrity checks, and contained only unencrypted regular files with safe paths, modest expansion, and no links or device entries. Inspected content included both attachment manifests; `NEXRAD_PROVENANCE.json`, `OUTPUT_CONTRACT.json`, `README_DATA.md`, `nexrad_public.npz`, all four runtime declarations, and `nowcast_starter.py`; plus the ALE readme, originality and provenance notes, task readme, task card, task hook, scorer, full provenance, evaluation contract, `nexrad_hidden.npz`, and all six expert-output files. No attachment content remained unreviewed. NumPy members and standalone arrays were inspected for headers, member inventories, types, shapes, ranges, finiteness, timestamps, and solver/reference separation. The review environment lacked the declared Python runtime, so submitted programs were not executed; static source inspection and independent array and metric parsing covered the material contracts. A contributor can use the staged public archive and pinned NumPy environment to select a causal method, generate the six artifacts, and receive objective evaluation after protected truth is loaded and deleted before isolated replay. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The protected future is deterministically derived from publicly distributed NOAA observations identifiable by station and event chronology. The risk is materially mitigated because the solver is fully offline, protected-source identifiers are omitted from solver-visible provenance, and hidden arrays are absent until evaluation. Live public-source availability was not externally verified under the review’s no-network boundary.\n\n**Recommended Changes:**\nThe submission provides unusually thorough provenance binding, causal isolation, replay security, and objective scientific verification. No changes are required.","reviewed_at":"2026-08-16T07:47:57.182746Z"},"claim":null}