{"claim":{"id":"79957133-b2fa-4d12-956d-ba9e86ae87d2","submission_id":"e6caff5d-ba04-428c-a46e-c1ece532e7cf","submission_revision_id":"d32221b8-a6ba-498a-afb4-4c0204ad350a","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:34:44.205767Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"e6caff5d-ba04-428c-a46e-c1ece532e7cf","submission_revision_id":"d32221b8-a6ba-498a-afb4-4c0204ad350a","revision_version":1,"submission_status":"under_review","submitted_at":"2026-09-30T15:19:57.579563Z","submitter_name":"Jixian Liu","submitter_email":"jliu376@jh.edu","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5.5","harness":"Claude Code","estimated_score":"20–40%"}},"submitted_at":"2026-09-30T15:19:57.579563+00:00","industry_domain":"Energy & Power Systems","task_short_name":"residual_load_dayahead_uq","operating_system":"Linux","task_description":"Objective:\nBuild the day-ahead forecasting system of a Belgian electricity market desk. Every day at 11:30 Brussels time (D-1) the desk needs a forecast of the Belgian residual load - Elia's total load minus PV minus wind generation, in MW - for every quarter-hour of the next local day D: point forecasts from a CNN and from an RNN, and 100 equally likely whole-day scenarios from a conditional diffusion model. The work has three stages: (1) formulate the problem correctly, (2) forecast with a CNN and an RNN, (3) quantify the uncertainty with a diffusion model. The system is then operated, one delivery day at a time, over the twelve months that follow the development data (2025-09-01 to 2026-08-31), which the agent never sees.\n\nStarting State:\nAn offline Ubuntu 22.04 VM (8 vCPU, no GPU, no network, sudo) with Python 3.10, PyTorch 2.11 (CPU), NumPy, pandas and SciPy.\n- input/TASK.md: the full brief, including every check and threshold. input/DATA.md: data dictionary with the publication time of every column. input/task_contract.json: machine-readable contract. input/examples/.\n- input/raw/: real Elia Open Data exports - total load; PV with 14 nested region rows per quarter-hour (Belgium, 3 regions, 10 provinces); wind in 5 disjoint parts - with measured values and Elia's day-ahead forecasts, plus GFS weather forecasts for five Belgian cities from the Open-Meteo archive. Local days 2021-04-01 to 2025-08-31, quarter-hourly, published text unchanged, 314 MB (CC BY 4.0).\n- input/calendar/: Belgian public holidays 2021-2026.\n- software/opsim/: the operator's simulator. It builds the as-of data package of any day, runs a system exactly as the evaluator does (including the DDPM sampling loop), scores runs and self-checks a system.\n- output/ is empty.\n\nInstructions & Rules:\nStage 1 - formulation.\n(a) Implement the target from the raw exports: residual_load = totalload - PV of Belgium - all Belgian wind (measured values). It is undefined when the load or any wind part is missing; a missing PV value counts as 0 MW at night (when the PV 11:00 forecast is 0).\n(b) A delivery day is a Europe/Brussels calendar day: 96 quarter-hours, 92 on the spring and 100 on the autumn clock-change day, reported with their UTC starts.\n(c) Use only what was published at D-1 11:30 (DATA.md). The operator passes the system an as-of package with unpublished values blank, and checks that filling them in changes nothing.\n(d) Write formulation.json (time zone, issue time, intervals per day, and the latest available interval of each of the 35 value columns) and formulation.md (sections Target, Delivery grid, Information set, Data quality, CNN, RNN, Diffusion, Backtest; at least 500 words).\nStage 2 - two complete point forecasters, both exported with torch.export: a CNN (convolutions, no recurrent or attention layer) and an RNN (LSTM/GRU/RNN layers, no convolution or attention). The feature code may transform the data but must not contain another fitted forecasting model.\nStage 3 - a DDPM epsilon-prediction denoiser exported with torch.export, plus its noise schedule (at most 250 steps). The operator runs its own sampling loop: scenario = base (the CNN, the RNN or their mean) + scale_mw x x0.\nDeliverables:\n- output/system/: SYSTEM.json, formulation.json, formulation.md, a Python package with two commands - `prepare` (as-of package -> model inputs) and `target` (raw files -> residual load) - and models/*.pt2.\n- output/backtest/: an opsim run of 2025-06-01 to 2025-08-31 made with the final system, plus metrics.json from opsim score.\nRules:\n- No network. Do not modify input/, software/ or the Python installation. Everything the system needs must be inside output/system/.\n- prepare: at most 60 s for any day and 60 min for the whole year.\n- Inference: at most 240 CPU-seconds for any day and 2 h for the whole year.\n- Time budget: 8 hours.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The evaluator (included: evaluation/main.py and evaluation/scripts/grader/) runs on the VM as root after the agent has finished. It only scores results it computed itself, against a hidden reference that is staged only after the agent is done.\n\nGates (score 0 if either fails):\n- The Python stack is unmodified: every installed torch/numpy/pandas file matches its pip RECORD hash, and dpkg --verify passes.\n- output/ contains no links or special files, and SYSTEM.json is valid.\n\nRe-execution:\n- The evaluator builds the as-of package of every hidden delivery day (2025-09-01 to 2026-08-31, 365 days) with the public simulator's own code.\n- It runs the submitted `prepare` on each package as an unprivileged user without network, one day at a time in date order.\n- It re-runs `prepare` on 8 probe days with every unpublished value filled in, and re-runs 6 random days of the submitted backtest.\n- A second unprivileged user executes the exported CNN, RNN and denoiser with the operator's own DDPM sampling loop (100 scenarios per day, evaluator seed).\n- After all forecasts are made, a third user runs the submitted `target` command on the development and hidden raw files.\nEverything is compared with the hidden official residual-load series.\n\nChecks (weights in brackets):\nS1 formulation:\n- target reproduced on both raw sets: at most 2 quarter-hours off by more than 0.5 MW or wrongly defined/undefined [2]\n- exact delivery grid on every day, including 2025-10-26 (100 quarter-hours) and 2026-03-29 (92) [1.5]\n- identical prepared inputs on all 8 future-filled probe days [2]\n- at least 99% of the days run through within the time limits [1]\n- formulation.json fully correct: issue time, intervals per day, latest available interval of all 35 value columns [1]\n- formulation.md has the required sections and at least 500 words [0.5]\nS2, for the CNN and for the RNN separately:\n- graph audit: CNN has convolutions and no recurrent/attention op; RNN has a recurrent op and no convolution/attention; at least 1000 parameters in those layers [0.5 each]\n- MAE over the hidden year <= 450 MW, i.e. better than Elia's own day-ahead forecast of the residual load, issued 6.5 h later (453 MW) [2 each]\n- MAE in every season <= 520 MW [1 each]\n- MAE on holidays, bridge days and clock-change days <= 470 MW [1 each]\n- |bias| <= 80 MW [0.5 each]\n- backtest evidence [1]: made with the submitted system (hash), metrics.json reproduces with the official target, re-run days give the same forecasts within 0.5 MW\nS3, the diffusion scenarios:\n- at most 250 steps, at most 240 CPU-s per day, sampling works on every day [0.5]\n- mean CRPS <= 330 MW [2]\n- 80% interval coverage 0.77-0.83 and 95% coverage 0.92-0.98 [1.5]\n- 80% coverage within 0.72-0.88 in every season [1]\n- 80% coverage within 0.72-0.88 in every tercile of the wind and of the PV day-ahead forecast [1.5]\n- energy score <= 0.99 x that of the same scenarios with the quarter-hours decoupled across members [1.5]\n- 80% coverage of the daily mean and of the daily maximum within 0.72-0.88 [1]\n- mean 80% interval width <= 1650 MW [0.5]\n- scenario metrics and re-runs of the backtest reproduce [0.5]\n\nScoring:\n- S1, S2, S3 = weighted fraction of each stage's passed checks.\n- Score = 0.3 x (0.2 x S1 + 0.35 x S2 + 0.45 x S3) + 0.7 x [every check of the three stages passed]. The desk needs one working system, so the completion share goes only to a submission that passes everything.\n\nReference and thresholds:\n- The included expert output scores 1.0 (all 28 checks passed) with the included evaluator on the full hidden year: CNN/RNN MAE 417.5/419.6 MW, CRPS 299, coverage 0.81/0.94 (expert_output/evaluator_report.json).\n- The same reference recipe, retrained from scratch inside an emulated offline Ubuntu 22.04 VM, also scored 1.0 in a full local ALE flow (start -> agent -> evaluate as root with the sandbox users).\n- The year-MAE limit is the industry bar (Elia's own later forecast); the other limits were calibrated on the reference's hidden year and on an independent calibration year (2024-09 to 2025-08). The record is in reference/protocol.json.\n- Weaker solutions that keep a perfect formulation and perfect point forecasts score 0.21-0.25: a Gaussian noise generator posing as the diffusion model, or a diffusion model trained on in-sample residuals. 13 negative controls fail exactly their intended checks.","software_and_version":"Python 3.10, PyTorch 2.11 (CPU, torch.export), NumPy 2.2.6, pandas 2.2.3, SciPy 1.15.3","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"765c93bf-3c23-4b7e-8684-eac334348231","filename":"inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":65251091},{"file_id":"4ea93df1-2a1e-41db-8109-a82880dba7ce","filename":"reference_and_evaluation.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":32021975}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally realistic, securely evaluable, and demonstrably reachable end to end on the declared offline CPU environment.\n\n**Completeness Check:**\n\n- Instruction: Complete — The brief precisely defines target construction, publication-time constraints, DST handling, model architecture requirements, output schemas, runtime limits, and deliverables.\n- Inputs: Complete — Realistic development data includes 154,944 quarter-hours, 14 PV regions, five disjoint wind parts, five weather locations, documented missingness, and a six-year holiday calendar.\n- Toolchain: Complete — The offline Python/PyTorch stack, public simulator, package metadata, version constraints, and CPU-only execution path are specified and mutually compatible.\n- Reference output: Complete — A full system, 92-day backtest, exported CNN/RNN/denoiser models, oracle training workflow, hidden truth, and an evaluator report passing all 28 checks are supplied.\n- Rubric: Complete — All gates, stage weights, thresholds, tolerances, resource limits, probe behavior, scoring formulas, and partial-credit rules are machine-readable and implemented by the evaluator.\n\n**File and Task Reachability:**\n\nThe attachment classification is `reachable`, and the end-to-end task classification is `reachable`. Both manifest attachments, `inputs.zip` and `reference_and_evaluation.zip`, were inspected. Their downloaded byte counts match the manifest; both are readable, unencrypted ZIP containers with valid compressed data, safe member names, no links or special files, and modest expansion sizes. The first contains 18 members and the second 244 members.\n\nInspection covered `TASK.md`, `DATA.md`, the task contract, examples, holiday data, all four development CSV schemas and representative records, simulator documentation and core source, environment metadata, the task card, evaluator and sandbox logic, protocol and formulation answers, hidden raw/truth/strata schemas, expert manifests and reports, the backtest manifest and metrics, one representative forecast NPZ, all three exported model containers and graph operators, and representative oracle training code. All JSON documents parsed successfully. The remaining 91 backtest NPZ payloads and individual model tensor values were inventoried and container-checked but not decoded value by value; uploaded code and models were not executed during review. Public-source and licensing claims are documented consistently but were not independently verified because external lookup is outside the evidence boundary.\n\nThe completion path is coherent: the contributor receives development data and the public simulator, builds the declared system and backtest, and the evaluator later stages protected reference material, verifies installation and output integrity, generates daily as-of packages, runs preparation under an unprivileged identity, executes exported models through evaluator-owned inference and DDPM code, runs leakage probes and target reconstruction, and scores against hidden truth. The reference report records 365/365 successful hidden days, exact grids and targets, passing accuracy/calibration thresholds, and a final score of 1.0. Hidden truth and oracle material are explicitly segregated and checked as unavailable before the contributor run.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\nNone\n\n**Recommended Changes:**\n\nThe submission already provides strong instructions, authentic-looking operational data, reproducible reference evidence, and comprehensive evaluation isolation. No changes are required.","reviewed_at":"2026-09-30T15:36:32.928833Z"},"claim":{"id":"79957133-b2fa-4d12-956d-ba9e86ae87d2","submission_id":"e6caff5d-ba04-428c-a46e-c1ece532e7cf","submission_revision_id":"d32221b8-a6ba-498a-afb4-4c0204ad350a","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:34:44.205767Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}